devices and methods of image file generation and image processing, non-transient computer-readable r

专利摘要:
DEVICES AND METHODS OF IMAGE FILE GENERATION AND IMAGE PROCESSING, COMPUTER PROGRAM, AND DATA STRUCTURE OF AN IMAGE FILE. In a data structure provided for image files, hierarchical data, which contains image data for a single image in different resolutions in order of resolution, uses a structure comprising a header (150), index blocks (160) and tiles of image (170). The header (150), which defines a plurality of regions that divide a pyramidal hierarchical structure into a virtual space that generates the hierarchical data, consists of a pointer pointed at one of the index blocks (160) for each region. The index blocks (160), which are generated for each region in the structure defined in the header (150), consist of pointers pointed at tiles of an image (170) for each position in multilevel images that belong to each region. The image tiles (170) are the image data used during the actual rendering of the image.
公开号:BR112012009893B1
申请号:R112012009893-4
申请日:2010-07-29
公开日:2021-02-02
发明作者:Tetsugo Inada
申请人:Sony Computer Entertainment Inc.；
IPC主号:

专利说明:

Field of the Invention
[001] The present invention relates to an image processing technology to enlarge / reduce an image displayed on a screen or to move the image up, down, left or right. Background of the Invention
[002] Home entertainment systems capable of reproducing moving images as well as running game programs are proposed. In home entertainment systems, a GPU generates three-dimensional images using polygons (see, for example, patent document 1).
[003] Regardless of the purpose of image display, improving image display efficiency is an important and constant challenge. High-speed rendering of high-definition images requires several efforts. For example, a method of storing texture data separately is proposed to make the mapping process efficient (for example, non-patent documents 1 and 2).
[004] Patent document 1 - US patent 6,563,999.
[005] Non-Patent Document 1 - Sylvain Fefebvre, et. al., Unified Texture Management for Arbitrary Meshes, Repport de recherche, N5210, May 2004, Institut National De Recherche En Informatique Et En Automatique.
[006] Non-Patent Document 2 - Martin Kraus, et. al., Adaptive Texture Maps, Graphics Hardware (2002), pages 1-10, The Eurographics Association SUMMARY OF THE INVENTION PROBLEM TO BE SOLVED BY THE INVENTION
[007] For responsive display of high definition images, reduction in data size and high speed rendering are important and constant challenges. When a need arises to update a part of an image, complex processes, for example, to fully update the data structure once built are likely to be required, even if the aforementioned approach to designing a data structure (for example, texture mapping) is used.
[008] The present invention addresses the problem, and a purpose of this is to provide an image processing technology capable of reducing the data size of a large image and, at the same time, efficiently generate, display and modify the image. MEANS TO SOLVE THE PROBLEM
[009] One embodiment of the present invention concerns an image file generation device. The image file generation device comprises: an image hierarchy generation unit configured to generate a plurality of parts of image data with different resolutions from input image data and to generate hierarchical data formed by hierarchizing the plurality of data pieces according to the resolution; an image splitting unit configured to divide image data into each layer of hierarchical data into tiled images of a predetermined size; an index creation unit configured to create index data that map tile locations defined by dividing an image into each layer of hierarchical data in tile images to data for the tile images used to render the tile locations; and an image file generation unit configured to generate an image file that includes data for the tiled images and the index data, where the index block creation unit creates the index data in the form of index that organize index data for respective tile locations, for each of the areas produced by the division of a hierarchical structure that comprises the hierarchical data in a virtual space defined by an image plane of the hierarchical data and a geometric axis of resolution perpendicular to the image plane.
[0010] Another embodiment of the present invention concerns an image processing device. The image processing device comprises: a storage device configured to store an image file that includes data for tile images produced by dividing, in a predetermined size, images into hierarchical data that comprise image data representing a single image at different resolutions, and that includes index data that map tile locations defined by dividing an image into each layer from hierarchical data in tile images to data for tile images used to render tile locations; an input information acquisition unit configured to recognize a user request to change a display area that includes movement in an image plane and to change the resolution; and a tile image identification unit configured to identify data for a tile image needed to render an area that must be displayed again according to the request to change a display area by reference to the index block; a display image processing unit configured to read the data for the identified tile image from the storage device and render the area that must be displayed again, where the index data is in the form of index blocks that organize index data for respective tile locations, for each of the areas produced by dividing a hierarchical structure comprising hierarchical data in a virtual space defined by an image plane of the hierarchical data and a geometric axis of resolution perpendicular to the image plane .
[0011] Another embodiment of the present invention concerns an image processing device. The image processing device comprises: an image file acquisition unit configured to read an image file that includes data for tile images produced by dividing, in a predetermined size, images into hierarchical data that comprise image data that represent a single image in different resolutions, and that includes index data that map tile locations defined by dividing an image into each layer of hierarchical data in tile images to data for the tile images used to render the tile locations; an update information acquisition unit configured to acquire update information on an image displayed by using the image file; a tiled imaging unit configured to generate data for a new tiled image used to render an area to be updated based on the update information; an index data update unit configured to update data for a tile image mapped to a tile location defined in the index data as included in the area to be updated in the data for the new tile image; and an image file generation unit configured to generate an image file that includes data for the tile image generated by the tile image generation unit and the index data updated by the index data update unit, where the index data is in the form of index blocks that organize index data for respective tile locations, for each of the areas produced by the division of a hierarchical structure that comprises the hierarchical data in a virtual space defined by an image plane of the hierarchical data and a geometric resolution axis perpendicular to the image plane.
[0012] Another embodiment of the present invention concerns an image file generation method. The image file generation method comprises: generating a plurality of pieces of image data with different resolutions from input image data, generating hierarchical data formed by hierarchizing the plurality of pieces of data according to the resolution and storing the data in a memory; read image data in each layer of hierarchical data from memory and divide the read data into tile images of a predetermined size; create index data that map tile locations defined by dividing an image into each layer from hierarchical data in tile images to data for tile images used to render tile locations and store index data in memory; and generate an image file that includes data for the tiled images and the index data and store the generated file in memory, where the creation of index data creates the index data in the form of index blocks that organize data from index for respective tile locations, for each of the areas produced by dividing a hierarchical structure that comprises the hierarchical data in a virtual space defined by an image plane of the hierarchical data and a geometric axis of resolution perpendicular to the image plane.
[0013] Another embodiment of the present invention concerns an image processing method. The image processing method comprises: reading, from a memory, an image file that includes data for tile images produced by the division, in a predetermined size, of images in hierarchical data that comprise image data that represent a single image in different resolutions, and which includes index data that map tile locations defined by dividing an image into each layer from hierarchical data in tile images to data for the tile images used to render the tile locations; recognize a user request to change a display area that includes movement in an image plane and to change the resolution; identify data for a tile image needed to render an area that must be displayed again according to the request to change a display area by reference to the index block; and read the data for the identified tile image from memory, render the area that must be displayed again and display the rendered data on a display device; where the index data is in the form of index blocks that organize index data for respective tile locations, for each of the areas produced by dividing a hierarchical structure comprising the hierarchical data in a virtual space defined by a plan of image of hierarchical data and a geometric axis of resolution perpendicular to the image plane.
[0014] Another embodiment of the present invention relates to an image processing method. The image processing method comprises: reading, from a memory, an image file that includes data for tile images produced by the division, in a predetermined size, of images in hierarchical data that comprise image data that represent a single image in different resolutions, and which includes index data that map tile locations defined by dividing an image into each layer from hierarchical data in tile images to data for the tile images used to render the tile locations; acquire information about the update on an image displayed by using the image file; generate data for a new tile image used to render an area to be updated, based on the update information, and store the data generated in memory; update data for a tile image mapped to a tile location defined in the index data as included in the data area for the new tile image; and generate an image file that includes data for the tiled image generated by the tiled imaging unit and the index data updated by the index data update unit, and store the data in memory, where the data of index are in the form of index blocks that organize index data for respective tile locations, for each of the areas produced by dividing a hierarchical structure that comprises hierarchical data in a virtual space defined by an image plane of hierarchical data and a geometric axis of resolution perpendicular to the image plane.
[0015] Another embodiment of the present invention concerns a data structure of an image file. The data structure is an image file read from a storage device to display at least part of an image on a screen. The data structure maps data to tile images produced by dividing, at a predetermined size, image data representing a single image at different resolutions, in index data, the index data mapping tile locations produced by dividing a image in each layer of hierarchical data in tiled images up to data for the tiled images used to render the tile locations, the hierarchical data being formed by the hierarchy of said image data according to the resolution, and in which the data of index are in the form of index blocks that organize index data for respective tile locations, for each of the areas produced by dividing a hierarchical structure that comprises hierarchical data in a virtual space defined by an image plane of hierarchical data and a geometric axis of resolution perpendicular to the image plane.
[0016] Another embodiment of the present invention concerns an image processing device. The image processing device comprises: an image processing request unit configured to run a program and issue an image processing request that includes information required to render an image that must be displayed again; a hierarchical data storage unit configured to store a hierarchical color buffer and a hierarchical Z buffer that hierarchize a pixel plane of the color value that stores color information for pixels of an image to be rendered and a pixel plane of Z value that stores information about the depth of pixels from a viewing point, respectively, according to a plurality of resolutions of the image to be rendered; an image processing unit configured to recognize the image processing request and perform image processing to update the hierarchical color temporary storage and the hierarchical Z temporary storage, where the image processing unit comprises: a color temporary storage rendering and Z rendering temporary storage that store data for a layer and area to be rendered, the data being read from the hierarchical color temporary storage and the hierarchical Z temporary storage; a rendering unit configured to identify a layer and area to be rendered by reference to the image processing request, loading associated data into the rendering color buffer and the Z rendering buffer, and performing computation for image processing, and update the data according to the image processing request; and a hierarchical data update unit configured to scale the data in the rendering color buffer and the rendering Z buffer as rendered, adapting to the plurality of resolutions, to update layered data from the hierarchical color buffer and the hierarchical Z temporary storage.
[0017] Another embodiment of the present invention concerns an image processing method. The image processing method comprises: running a program and issuing an image processing request that includes information required to render an image that must be displayed again; store a hierarchical color temporary store and a hierarchical Z temporary store that hierarchize a pixel plane of the color value that stores color information for pixels of an image to be rendered and a pixel plane of Z value that stores depth information of pixels from a viewing point, respectively, according to a plurality of resolutions of the image to be rendered; and recognize the image processing request and perform image processing, to update the hierarchical color temporary storage and the hierarchical Z temporary storage, where the update of the hierarchical color temporary storage and the hierarchical Z temporary storage comprises: identifying a layer and an area to be rendered by reference to the image processing request, loading associated data into the hierarchical color temporary storage and the hierarchical Z temporary storage, and writing the data to the rendering color temporary storage and the Z rendering temporary storage; perform computation for image processing by reference to the image processing request to update the rendering color buffer and the Z rendering buffer; and scale the data in the rendering color buffer and the Z rendering buffer as updated, adapting to the plurality of resolutions, to update layered data from the hierarchical color buffer and the hierarchical Z buffer.
[0018] Optional combinations of the aforementioned constituent elements and implementations of the invention in the form of methods, apparatus, systems, computer programs, data structures and recording media can also be practiced as additional modes of the present invention. ADVANTAGE OF THE PRESENT INVENTION
[0019] The present invention provides an image processing device capable of efficiently constructing image data to display images in a wide variety of resolutions. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 shows an environment in which an information processing system according to the first modality is used; figure 2 shows the appearance of the input device that can be used in the image processing system of figure 1; figure 3 shows the hierarchical structure of the image data used in the first modality; figure 4 shows the configuration of the information processing device according to the first embodiment; Figure 5 schematically shows the image data flow according to the first embodiment; figure 6 shows a process of pre-fetching image data according to the first modality; figure 7 shows the configuration of a control unit with the function of displaying hierarchical data according to the first modality with details; figure 8 shows schematically the data structure of the hierarchical data used in the first modality; figure 9 shows the configuration of a control unit with the function of generating an image file for display in accordance with the first modality; figure 10 shows an exemplary relationship according to the first modality between an original image and tiled images in the presence of redundancy in an image; figure 11 shows another exemplary relationship according to the first modality between an original image and tiled images in the presence of redundancy in an image; figure 12 shows a method for defining, in the header, how a tile image is shared in the presence of redundancy according to the first modality; figure 13 shows a method for defining, in the index block, how tiled images are shared in the presence of redundancy according to the first modality; figure 14 is a flow chart showing steps according to which the control unit shown in figure 9 generates an image file; figure 15 is a flow chart showing steps according to which the control unit displays an image file according to the first modality; figure 16 shows the configuration of a control unit with the function of modifying an image according to the first modality; figure 17 shows an explanatory example of modifying an image according to the first modality; figure 18 schematically shows how pointers in the index block change in the modification shown in figure 17; figure 19 shows steps for adding a layer of a higher resolution according to the first embodiment; figure 20 illustrates how the header and the index block change according to the first modality when there were no index blocks to which the layer to be added belongs; figure 21 shows steps for adding a new area to an existing image according to the first modality; figure 22 shows how the header and index blocks change when a new area is added according to the first modality; figure 23 shows how the division into areas is defined in the header when a layer is added at the top of the hierarchical structure according to the first modality; figure 24 is a flow chart showing steps taken by the image processing device when the user modifies or revises an image according to the first modality; figure 25 shows the structure of the control unit and the main memory of the information processing device according to the second embodiment with additional details; figure 26 is a flowchart showing steps for rendering hierarchical data according to the second modality; figure 27 shows the configuration of the device, in which a scheduler to perform a scheduling process is built in the main memory according to the second modality; figure 28 shows the addressing in a GPU temporary storage according to the second modality; and figure 29 shows the addressing in a GPU temporary storage according to the second modality. BEST MODE FOR CARRYING OUT THE INVENTION First Mode
[0021] Image data subject to processing according to this modality has a hierarchical structure that comprises images of different resolutions generated by reducing the size of an original image in a plurality of stages. One image on each layer is divided into a tiled image or a plurality of tiled images. For example, the image with the lowest resolution comprises a tiled image. The original image with the highest resolution comprises the largest number of tiled images. Images are displayed in such a way that an enlarged view or a reduced view is presented efficiently by switching from an image currently used for rendering to a tile image of a different layer when the displayed image reaches a predetermined resolution.
[0022] A description of the basic display mode of an image with a hierarchical structure such as the one described above will be given. Figure 1 shows an environment in which an information processing system 1 according to an embodiment of the present invention is used. The information processing system 1 comprises an information processing device 10 configured to run applications that include an image processing program and a display device 12 configured to transmit a processing result by the information processing device 10. The device display 12 may be a television set provided with a screen for transmitting an image and a loudspeaker for transmitting sound.
[0023] The display device 12 can be connected to the information processing device 10 by cable or connected wirelessly using, for example, wireless LAN (Local Area Network). The information processing device 10 of the information processing system 1 can be connected to an external network, such as the Internet, by a cable 14 and to transfer and acquire content including hierarchical compressed image data. The information processing device 10 can be connected wirelessly to an external network.
[0024] The information processing device 10 changes a display area by enlarging / reducing an image displayed on the screen of the display device 12 or by moving the image up, down, left or right, according with a user request. When the user manipulates an input device by viewing an image displayed on the screen, the input device transmits a request signal to change a display area to the information processing device 10.
[0025] Figure 2 shows the appearance of the input device 20. The input device 20 is provided with directional keys 21, analog levers 27a, 27b and four control buttons 26, which comprise the device for user control. The four buttons 26 comprise a circle button 22, a cross button 23, a square button 24 and a triangle button 25.
[0026] The user control device of the input device 20 of the information processing system 1 is assigned the function of inserting a request for enlarging / reducing a displayed image and inserting a request for scrolling up, down , left or right. For example, the function of inserting a request for enlarging / reducing a displayed image can be allocated on the right analog stick 27b. The user can enter a request to reduce a displayed image by pulling the analog stick 27b towards the user and can enter a request to enlarge a displayed image by pushing it away from the user. The function of inserting a movement request from a display area can be allocated to the directional keys 21. By pressing the directional keys 21, the user can enter a movement request in the direction in which the directional keys 21 are pressed. The function of inserting a request to change an image can be allocated on the alternate user control device. For example, the function of inserting a scroll request can be allocated on the analogue lever 27a.
[0027] In order to implement the functions described below, the input device 20 is additionally allocated the function of moving the displayed cursor in an image or selecting a file or a command. Input device 20 can be implemented by an ordinary input device, such as a pointing device, mouse, keyboard or touch panel. The allocation of the aforementioned functions can be determined appropriately, depending on the type of the input device 20.
[0028] The input device 20 has the function of transferring an input signal that requests a change of a display area to the information processing device 10. In the modality, the input device 20 is configured to be able to communicate wirelessly with the information processing device 10. The input device 20 and the information processing device 10 can establish communication using the Bluetooth protocol (registered trademark) or the IEEE802.11 protocol. The input device 20 can be connected to the information processing device 10 by means of a cable to transfer a signal that requests a change from a display area to the information processing device 10 in this way.
[0029] Figure 3 shows the hierarchical structure of the image data used in the modality. The image data has a hierarchical structure comprising a 0 th layer 30, a first layer 32, a second layer 34 and a third layer 36 in the depth direction (geometric axis Z). Although the figure shows only four layers, the number of layers is not restrictive. In the following, image data with a hierarchical structure like this will be referred to as "hierarchical data". However, it is clear that the hierarchical data illustrated in the figure represent merely a conceptual model. In practice, a plurality of data sets are used to represent hierarchical data.
[0030] The hierarchical image data shown in figure 3 have a quadruple tree hierarchical structure. Each layer comprises one or more tiled images 38. All tiled images 38 are formed to be the same size with the same number of pixels. For example, an image includes 256 x 256 pixels. The image data in the respective layers are representations of an image in different resolutions. The original image in the third layer 36 with the highest resolution is reduced in a plurality of stages to generate the image data for the second layer 34, the first layer 32 and the 0 th layer 30. For example, the resolution in N -th layer (N is an integer equal to or greater than 0) can be half the resolution of (N + 1) -th layer both in the horizontal direction (geometric axis X) and in the vertical direction (geometric axis Y).
[0031] In information processing device 10, hierarchical image data is compressed in a pre-defined compression format, is stored on a storage device, is read from the storage device and decoded before being displayed on the screen. The information processing device 10 according to the modality is provided with the decoding function compatible with a plurality of compression formats. For example, the device is capable of decoding compressed data in S3TC format, JPEG format, JPEG2000 format.
[0032] As shown in figure 3, the hierarchical structure of the hierarchical data is configured in such a way that the horizontal direction is defined along the geometric axis X, the vertical direction is defined along the geometric axis Y and the direction of the depth be defined along the geometric axis Z, thereby building a virtual three-dimensional space. By deriving the amount of change in the displayed image by referring to the signal supplied from the input device 20 and requesting change of a display area, the information processing device 10 uses the amount of change to derive the coordinates at the four corners of a frame (frame coordinates) in the virtual space. Frame coordinates in the virtual space are used to load compressed data into main memory or to generate a displayed image. Instead of the frame coordinates in the virtual space, the information processing device 10 can derive information that identifies the layer and texture coordinates (UV coordinates) in the layer. In the following, the combination of information identifying the layer and texture coordinates will also be referred to as the frame coordinates.
[0033] Figure 4 shows the configuration of the information processing device 10. The information processing device 10 comprises an air interface 40, a switch 42, a display processing unit 44, a hard disk drive 50, a recording media loader 52, a disk unit 54, a main memory 60, a temporary storage memory 70 and a control unit 100. The display processing unit 44 is provided with a frame memory to store temporarily data to be displayed on the display device screen 12.
[0034] Switch 42 is an ETHERNET switch (ETHERNET is a registered trademark), a device connected by cable or wirelessly to an external device to transmit and receive data. Switch 42 is connected to an external network via cable 14 to receive data, for example, content from a server. The switch 42 is connected to the air interface 40. The air interface 40 is connected to the input device 20 using a pre-defined wireless communication protocol. A signal entered by the user via the input device 20 is supplied to the control unit 100 via the air interface 40 and the switch 42.
[0035] The hard disk drive 50 acts as a storage device for storing data. The hierarchical image data received via switch 42 is stored on hard drive 50. When a removable recording medium, such as a memory card, is mounted, the recording medium loader 52 reads data from the removable recording media. When a ROM disk is mounted, disk unit 54 drives and recognizes the ROM disk to read data. The ROM disk can be an optical disk or a magneto-optical disk. Hierarchical image data can be stored on the recording medium.
[0036] The main controller 100 is provided with a multi-core CPU. A general-purpose processor core and a plurality of simple processor cores are provided on a single CPU. The general-purpose processor core is referred to as a power processing unit (PPU) and the other processor cores are referred to as synergistic processing units (SPU).
[0037] The main controller 100 is provided with a memory controller connected to main memory 60 and temporary storage memory 70. The PPU is provided with a register and a main processor as an executing entity. The PPU efficiently allocates tasks as basic processing units in applications in the respective SPUs. The PPU itself can perform a task. The SPU is provided with a register, a subprocessor as an executing entity and a local memory as a local storage area. Local memory can be used as temporary storage memory 70.
[0038] Main memory 60 and temporary storage memory 70 are storage devices and are formed as random access memories (RAM). The SPU is provided with a dedicated direct memory access controller (DMA) and is capable of transferring data at high speed between main memory 60 and temporary storage memory 70. High-speed data transfer is also achieved between memory frame in display processing unit 44 and buffer memory 70. Control unit 100 according to the embodiment implements high-speed image processing by operating a plurality of SPUs in parallel. The display processing unit 44 is connected to the display device 12 and transmits an image processing result according to the user's request.
[0039] The information processing device 10 according to the modality is configured to pre-load part of the compressed image data from the hard disk drive 50 into the main memory 60 in order to change an image displayed smoothly as the image displayed is enlarged / reduced or the display area moves. In addition, device 10 is configured to pre-decode part of the compressed image data loaded into main memory 60 and store the decoded data in temporary storage memory 70. This allows instant switching of the images used to create the displayed image when switching is required posteriorly.
[0040] Figure 5 schematically shows the flow of image data according to the modality. Hierarchical data is stored on hard drive 50. A recording medium mounted on the recording media loader 52 or disk drive 54 can store data instead of hard drive 50. Alternatively, the information processing device 10 it can transfer hierarchical data from an image server connected to the information processing device 10 via the network. As explained, hierarchical data is compressed in a fixed-length format, such as S3TC, or in a variable-length format, such as JPEG.
[0041] From the hierarchical data, part of the image data is loaded into main memory 60, maintaining a compressed state (S10). An area to be loaded is determined according to a predefined rule. For example, an area close to the image currently displayed in the virtual space or an area predicted as frequently requested for display, from the point of view of the image content or browsing history by the user, is loaded. The data is loaded not only when a request to change an image is originated, but also at predefined time intervals. This prevents heavy traffic for loading processes from occurring in a short period of time.
[0042] From the compressed image data stored in main memory 60, data for an image of an area required for display or data for a tile image of an area deemed necessary is decoded and stored in temporary storage memory 70 (S12) . The temporary storage memory 70 includes at least two temporary storage areas 72 and 74. The size of the temporary storage areas 72 and 74 is configured to be larger than the size of the frame memory 90, so that when the inserted signal by means of the input device 20 requesting a change, in the display area, of a certain degree or less, the image data loaded in the temporary storage areas 72 and 74 are sufficient to create a displayed image.
[0043] One of the temporary storage areas 72 and 74 is a temporary display storage used to store an image for creating the displayed image and the other is a decoding temporary storage used to make an image available to become necessary subsequently. In the example in figure 5, the temporary storage area 72 is a display temporary storage, the temporary storage area 74 is a decoding temporary storage, and a display area 68 is being displayed. The image stored in the decoding buffer in a prefetch process described below can be the same layer as the image stored in the display buffer or a different layer with a different scale.
[0044] Of the images stored in the temporary storage area 72, that is, the temporary display storage, the image of the display area 68 is rendered in frame memory 90 (S14). In the meantime, the image of a new area is decoded as needed and stored in the temporary storage area 74. The display temporary storage and decoding temporary storage are switched depending on the storage completion timing or the amount of change of the storage area. display 68 (S16). This allows for smooth switching between images displayed in the event of movement of a display area or scale change.
[0045] Figure 6 shows a prefetch process. Figure 6 shows the hierarchical data structure. The layers are represented as L0 (0- th layer), L1 (first layer), L2 (second layer) and L3 (third layer), respectively. In the hierarchical structure of the data shown in figure 6, the position in the direction of depth (geometric axis Z) indicates the resolution. The closer to L0, the lower the resolution, and the closer to L3, the higher the resolution. In terms of the size of the image displayed on the screen, the position in the direction of the depth represents the scale. Whereas the scale of the image displayed in L3 is 1, the scale in L2 is 1/4, the scale in L1 is 1/16 and the scale in L0 is 1/64.
[0046] Therefore, if the displayed image changes in the direction of the depth away from L0 in the direction of L3, the displayed image is enlarged. If the displayed image changes in the direction away from L3 in the direction of L0, the displayed image is reduced. An arrow 80 indicates that a sign requesting a change in the display area entered by a user requesting a reduction in the displayed image and shows that the reduction occurs through the 1/4 scale (L2). In the information processing device 10, the position of L1, L2, which becomes available as tile images 38, in the depth direction, is defined as the prefetch limit in the depth direction. When a signal that requests a change in the image indicates crossing of the prefetch limit, the prefetch process is initiated.
[0047] When the scale of the displayed image is close to L2, the displayed image is created by using the tile image in L2 (second layer). More specifically, the L2 tile image is used when the scale of the displayed image is between a switching limit 82 and a switching limit 84, the limit 82 being between the L1 tile image and the L2 tile image, and the limit 84 being between the L2 tile image and the L3 tile image. Therefore, when the reduction of an image is requested, as indicated by an arrow 80, the enlarged version of the L2 tile image becomes a reduced version and is displayed. In the meantime, the information processing device 10 also identifies the tile image 38 which is expected to be needed in the future based on the signal requesting a change in the image and decodes the identified image. In the example in figure 6, when the reduction scale requested by the signal requesting a change in the display area exceeds L2, the image processing device 10 prefers the tile image 38 in L1, which is located in the reduction direction , from hard drive 50 or main memory 60, decodes the scanned image and writes the decoded image to temporary storage memory 70.
[0048] Although the exposed description concerns the process of prefetching in the direction of depth, prefetching in the directions up, down, left or right in the identical layer is also processed in a similar way. More specifically, the prefetch limit is defined in the image data stored in the temporary storage memory 70 so that when the display position indicated by the signal requesting a change in the image exceeds the prefetch limit, the process prefetch is started.
[0049] Figure 7 shows in detail the configuration of the control unit 100a with the function of displaying hierarchical data described above according to the modality. The control unit 100a comprises an input information acquisition unit 102 for acquiring information entered by the user via the input device 20, a tile image identification unit 110 for identifying a tile that includes an area that must be re-entered. a loaded block determination unit 106 for determining an image block that must be loaded again and a loading unit 108 for loading a required image block from the hard drive 50. The control unit 100a further comprises a decoding unit 112 for decoding compressed image data and a displayed image processing unit 114 for rendering a displayed image.
[0050] The elements represented in figure 7 as functional blocks to carry out various processes are implemented in hardware, such as a central processing unit (CPU), memory or other LSI's, and in software, such as programs, etc. loaded into memory. As explained, the control unit 100 includes a PPU and a plurality of SPUs. PPU and SPUs form the individual functional blocks or in combination. Therefore, it will be obvious to those skilled in the art that function blocks can be implemented in a variety of ways only by hardware, only by software or a combination of these.
[0051] The input information acquisition unit 102 acquires an instruction entered by the user through the input device 20 to start / end the display of an image, move the display area, enlarge or reduce the displayed image, etc. The tile image identification unit 110 identifies a tile image that includes an area that must be displayed, according to the frame coordinates of the current display area and the information about the request to change the display area entered by the user. If the tiled image is already loaded in main memory 60, the tile image identification unit 110 supplies the information obtained as a result of identification to the decoding unit 112. If not, the tile image identification unit 110 supplies the information. information to the determination unit of the loaded block 106. In addition to the image necessary to render the current displayed image, the tile image identification unit 110 can identify a tile image predicted as necessary subsequently.
[0052] The loaded block determination unit 106 identifies an image block that must be loaded again from the hard disk 50 into main memory 60 by reference to the information coming from the tile image identification unit 110 and issues a load request to the loading unit 108. As long as loading unit 108 is not carrying out the loading process, the loading unit determination unit 106 can issue a loading request according to a predetermined timing schedule. For example, the loaded block determination unit 106 can issue a load request at predetermined time intervals or when the user issues a request to change the display area. The loading unit 108 performs an actual loading process according to a request from the determination unit of the loaded block 106.
[0053] The decoding unit 112 reads and decodes the data for the tile image from main memory 60 by reference to the tile image information acquired from the tile image identification unit 110 and stores the decoded data decoding temporary storage or display temporary storage. The displayed image processing unit 114 queries the frame coordinates of the newly displayed image, reads the corresponding image data from the display buffer in the buffer memory 70 and renders the data in the frame memory of the image processing unit. display 44.
[0054] Now, a description will be given of a modality adapted to the technology of displaying image data with the exposed hierarchical structure, and capable of building and transmitting image data more efficiently. Figure 8 shows schematically the data structure of the hierarchical data according to the modality. The hierarchical data according to this modality comprises three types of data, including a header 150, index blocks 160 and tile images 170. Header 150 and index blocks 160 are indexes for identifying a tile image subject to processing with based on a position in the virtual space formed by hierarchical data. In this modality, the hierarchical structure in the virtual space is divided into areas and is organized area by area using indexes that point to tile image data to enable efficient access to tile images.
[0055] Header 150 defines a plurality of areas produced by dividing the pyramidal hierarchical structure in virtual space, as shown in figure 3. A pointer that points to one of the index blocks 160 is defined in each area. In relation to the header 150 shown in figure 8, a triangle 151 represents the shape of the hierarchical structure viewed from the side. Broken horizontal lines indicate images 154a, 154b, 154c, 154d, 154e, ... in the respective layers. The hierarchical structure is divided into areas indicated by solid lines to define areas that include an area 152a, an area 152b, an area 152c, etc. Basically, the header 150 is configured in such a way that a pointer that points to one of the blocks index 160 is defined for each area.
[0056] Figure 8 shows that a pointer (arrow A) that points to an index block 160a is defined for area 152a, a pointer (arrow B) that points to an index block 160b is defined for area 152b, a pointer (arrow C) pointing to an index block 160c is defined for area 152c. As stated, however, a null pointer that does not point to any specific index block can be defined for some of the areas defined in the header.
[0057] The index blocks 160 are generated for the respective areas in the structure defined by the header 150. A pointer that points to one of the tile images 170 is defined for a position in the images in a plurality of layers that belong to each area. As shown in figure 8, an index block corresponding to an area in the header 150 includes data for at least some of the areas in the images in the plurality of layers that belong to the area. In the illustrated example, the index block 160a corresponding to area 152a contains the entire data for images 154a and 154b in the two layers that belong to area 152a. The index block 160b corresponding to area 152b contains the entire data for images 154c, 154d and 154e in the three layers that belong to area 152b.
[0058] The index block 160c corresponding to area 152c contains the data for parts of the images 154f, 154g and 154h in the three layers that belong to area 152c. Each tile location defined by dividing an image plane into each layer in sizes of a tile image contains a pointer that points to one of the tile images 170, i.e., actual image data of the tile location. However, a tile location can contain pointers that point to a plurality of tile images if a tile location is configured to be rendered using a plurality of information items, as will be described below. The three planes shown in each of the index blocks 160a, 160b and 160c in figure 8 correspond to the layers that belong to the corresponding area. Each of the small rectangles in the layer represents a tile location.
[0059] Figure 8 shows that a pointer (arrow D) pointing to a tile image 170a is defined for a tile location 164a on index block 160b and a pointer (arrow E) pointing to a tile image 170b is set to a tile location 164b. A pointer that points to a specific tile image may not be defined for some of the tile locations defined in the index block, as described below. Instead, a null pointer can be defined. Tile images 170 represent compressed data for tile images. By configuring the hierarchical data, as shown, the tile image identification unit 110, having learned the frame coordinates in the virtual space of an image frame to be displayed, is able to identify a tile image 170 included in the frame of the image. image, proceeding through header 150 and then through index blocks 160.
[0060] Figure 8 shows that each of the index blocks 160a, 160b and 160c is shown containing index data (that is, pointers pointing to tile images 170) for image layers in three layers comprising, respectively, locations of tiles 4 x 4, 8 x 8 and 16 x 16. Ensuring that the number of tile locations that can be defined by a single index block 160 is consistent, the size of a storage area that is required to be reserved in main memory 60 when one of the index blocks 160 is read from the hard disk 50 remains unchanged, regardless of which index block is read. Consequently, the management of a storage area is made easier.
[0061] Therefore, areas defined in header 150 are preferably generated in such a way that the index blocks 160 are the same size. If the hierarchical data has a quadruple tree hierarchical structure, as shown in figure 3, in which the 0 th layer contains a single tile image, the images in the three layers that belong to area 152b of figure 8, that is, the image 154c in the second layer, the image 154d in the third layer and the image 154e in the fourth layer, comprise tile locations 4 x 4, 8 x 8 and 16 x 16, respectively.
[0062] Therefore, ensuring that a single block of index 160 is able to define image areas that comprise 4 x 4, 8 x 8 and 16 x 16 tile locations, as shown, the data in full for the images in the three layers it can be accommodated in a single index block 160b without any modification. In other words, pointers that point to tiled images can be defined in such a way that the entire areas of each image 154c on the second layer, image 154d on the third layer and image 154e on the fourth layer in the hierarchical data are mapped. for the entire areas in each of the 0 th layer 162d, the first layer 162e and the second layer 162f of the index block 160b, respectively.
[0063] Meanwhile, area 152a in header 150 includes images 154a and 154b in the 0 th layer and the first layer, respectively, and image 154a in the 0 th layer comprises a single tile image and image 154b in the The first layer comprises 2 x 2 tile images. As such, the number of layers and the number of tile locations are less compared to the number of areas that can be defined by a single index block 160a. In a case like this, the image 154a in the 0 th layer in the hierarchical data is mapped to a part of the 0 th layer 162a in the index block 160a (for example, the tile location at the top left) and a pointer that points to a tiled image is defined in this way. Similarly, the image 154b in the first layer is mapped to a part of the first layer 162b in the index block 160a (for example, 2 x 2 tile images at the top left) and pointers that point to tile images are defined in this way. Null pointers are defined for the other tile locations in the index block 160a.
[0064] Area 152c in header 150 includes images 154f, 154g and 154h in the fifth layer, the sixth layer and the seventh layer, respectively. These images comprise a larger number of tile images than the number of tile locations that can be defined by a single index block 160. Thus, the image plane is divided as shown in figure 8 so that some of the areas of each layer is mapped to the 0 th layer, the first layer and the second layer in the index block 160c, respectively. The image plane is divided to ensure that the images in a plurality of layers that belong to each index block are representations of the same area of the image in different resolutions.
[0065] The division of the hierarchical structure shown in figure 8 is for illustrative purposes only. The division can be determined as appropriate, depending on the original hierarchical data structure or a desirable size of the data in an index block. The way in which hierarchical data is divided may change as the original image is updated.
[0066] Now, a description will be given of a modality to generate an image file that includes hierarchical data with the aforementioned data structure. This modality can also be implemented by a device configuration similar to that of the information processing system shown in figure 1 and the information processing device 10 shown in figure 4. Figure 9 shows the configuration of a control unit 100b with the function of generating an image file for display in accordance with this modality. The control unit 100b can be provided with the function for displaying the image, like the control unit 100a shown in figure 7, but the function is not illustrated in figure 9. The image processing device 10 provided with the control unit 100b only with the function shown in figure 9 can be provided separately from the information processing device that displays the display function.
[0067] The control unit 100b includes an image hierarchy generation unit 120 configured to read image data stored on the hard disk 50 and hierarchize the data, an image division unit 122 configured to divide the image into each layer in tiled images, a redundancy detection unit 124 configured to analyze the image in each layer and detect redundancy, a header / index block generation unit 126 configured to generate data for a header and an index block under consideration redundancy and an image file generation unit 128 configured to generate an image file that must ultimately be transmitted and which includes a tile image, a header and an index block.
[0068] The hierarchy image generation unit 120 reads, from the hard disk drive 50, data for an image for which a file must be created. The image data read can be a single image data item of a certain resolution. The image data subject to processing can be designated by the user via the input device 20, as shown. Alternatively, a request can be recognized from another functional block (not shown) that obtained the original image. Then, the image hierarchy generation unit 120 generates image data from a hierarchical structure that includes the original image, by reducing the data for the image read in stages defined by the respective resolutions to generate data for reduced images.
[0069] The image division unit 122 generates data for tiled images by dividing an image into each layer in predetermined sizes. The tile image data is stored in main memory 60. The position of each tile image in the original image is managed, for example, by assigning identification numbers to the tile images.
[0070] The redundancy detection unit 124 detects redundancy between images on the same layer or images from different layers, by analyzing the images in the respective layers. For example, redundancy on the same layer can occur when the same tile image data can be used in a plurality of tile locations. Redundancy across different layers can occur when an image produced by enlarging an image in a low resolution layer does not look very different from an image in a high resolution layer. Areas with redundancy, as described above, can be displayed by consistently using data for a given tile image and without maintaining data for individual tile images. This will make it possible to compress image data. A specific method will be described below.
[0071] The header / index block generation unit 126 creates data for the header and for the index block, as explained. In an image where redundancy is not detected, a pointer that points to one of the index blocks is defined for all areas defined in the header. In addition, for all tile locations defined in the index blocks, a pointer that points to a tile image is defined. In an area of the image with redundancy, data for a tiled image is shared by defining a null pointer in the header or in the index block.
[0072] The unit of generation of the image file 128 reads the data for the tile image pointed by the pointer defined in the index block and generates fundamental image data by the sequential concatenation of the data. The image file generation unit 128 forms the fundamental output data by generating an image file that includes the image data, the header and the index block. Thus, the size of the image data can be efficiently reduced by detecting redundancy between images and using data for a single tiled image to render a plurality of areas. The image data according to this modality comprises images with different resolutions. Therefore, data can be compressed at a high rate by using data for a given tile image to represent images from different layers.
[0073] Figure 10 shows how an original image and tiled images are related in the presence of redundancy. With reference to figure 10, an image 180a and an image 180b are images in two layers representing the same area. In other words, image 180a is obtained by reducing image 180b. The grids shown in images 180a and 180b indicate outlines to divide the image into tiled images. In the figure, identification numbers are assigned to the tile locations in images 180a and 180b and to tile images 170 for descriptive purposes.
[0074] An ellipse and a triangle are considered to be drawn on images 180a and 180b and the remaining areas are filled with a single color to represent the background. Data for the tile images 170 extracted from the areas in the images are stored in main memory 60, regardless of the initial arrangement of the images. The tile location "4" in image 180a, which is an area filled with a single color, is mapped to the tile image "4" extracted from it. In this case, a pointer that points to the tile image "4" is set to the tile location "4" in the index block 160.
[0075] Since the image 180b is a version of the image 180a with a higher resolution, the correspondence between tile locations in the two images can be easily derived based on the magnification factor. In the illustrated example, tile location "4" in image 180a is mapped to tile locations "15", "16", "19" and "20" in image 180b. Therefore, if tile location "4" in image 180a is filled with a single color, all tile locations "15", "16", "19" and "20" in image 180b are known to be filled with a single color. In other words, only the "4" tile image is required to display these areas.
[0076] Similarly, by scanning the image 180a to identify areas that will be filled with a single color when the image 180a is enlarged to the resolution of the image 180b, the bottom half of the tile location "3", that is, the tile locations tiles "17" and "18" in image 180b, are found to be filled with a single color. Therefore, these areas can also be displayed by using the "4" tile image. By sharing a tile image in this way, data for tile images can be compressed. Additionally, by not providing a specific pointer in an area in a higher resolution image that can be displayed using a lower resolution tile image, data for the header or for the index blocks can also be compressed.
[0077] According to this modality, representations of the same image in different resolutions are subject to processing. Therefore, by sequentially repeating the exposed redundancy detection process, starting with the image with the lowest resolution, areas with redundancy in a large, high-resolution image can be identified with minimal computational load. In the aforementioned modality, areas filled with a single color are detected so that data for a tile image is shared by the areas concurrently identified as filled with a single color. A similar process can also be used in areas virtually filled with a single color, a random pattern, or a pattern that comprises only repetition. The detection of these areas is performed using an area-by-area color histogram or frequency analysis, proceeding from a low resolution image.
[0078] Figure 11 shows how an original image and tiled images are related in the presence of redundancy according to another example. Images 180a and 180b are similar to those shown in figure 10. In relation to the figure, tile location "2" in image 180a is mapped to tile locations "7", "8", "11" and "12" in image 180b. Since the 180b image is of a higher resolution than the 180a image, the four tile images extracted from these areas generally include more information than a simple enlarged version of the "2" tile image extracted from the area "two".
[0079] However, there may be cases where the images in areas "7", "8", "11" and "12" in image 180b do not look different from an enlarged version of the "2" tile image, such as when the original image is a photo and the four areas are out of focus. In a case like this, as in the case of areas filled with a single color described above, an enlarged version of the tiled image "2" can be used to display areas "7", "8", "11" and "12" instead of the tiled images extracted from these areas, in such a way that the resulting image does not look different from the original image. Redundancy such as this is identified when the difference between an enlarged version of a lower resolution image and a higher resolution image is equal to or less than a threshold value, or when the frequency analysis of a high resolution image reveals only one frequency band as low as or below a threshold value.
[0080] Figure 12 shows a method for defining, in the header, how a tile image is shared in the presence of redundancy. Of the areas in the hierarchical structure defined in header 150a, pointers that point to specific index blocks are not defined for those areas that can be displayed using an image on a layer included in another area. Instead, null pointers are defined for these areas. In the example in figure 12, header 150a is constructed in such a way that pointers that point to specific index blocks, that is, valid pointers, are defined for the shaded areas, and null pointers are defined for the blank areas.
[0081] It is considered that an area of the image 156 in the hierarchical data with the header 150a is displayed. The figure shows that a null pointer is defined for the area that includes the image area 156. In this case, the virtual space of the hierarchical data is searched in the direction in which the image is reduced to identify an area that includes a reduced version of the area of image 156 and for which a valid pointer is defined (arrow F). Since a valid pointer is defined in an area 152d, as shown in the figure, data for the index block 160d designated by the pointer is acquired (arrow G).
[0082] The tile location in the second layer, the layer with the highest resolution among the layers for which the index block 160d contains data, which includes the image area 156 is identified. Then, the pointer mapped to the identified tile location is acquired. The location of the image area 156 in the image plane of the second layer of area 152d can be easily computed by reference to the scale of layers in the hierarchical data. The aforementioned tile location identification is made based on such computation.
[0083] Data for a tile image 170c designated by the acquired pointer is acquired (arrow H). The image displayed from the image area 156 is generated using the acquired data. Since the area of image 156 is the size of a tile image, a partial area 172 of tile image 170c, a reduced image, is enlarged as shown to render the area of image 156. As shown, the location of the area 172 corresponding to the area of the image 156 can be easily computed by reference to the scale of the area 172 in relation to the area of the image 156.
[0084] By allowing header 150a to define a null pointer, it is not necessary to provide index blocks for all areas defined by header 150a. As a result, the efficiency of data conversion increases. An area where a null pointer is defined need not be in the direction of the higher resolution. As shown in figure 12, a null pointer can be defined for an area 152e, which is sandwiched between a low resolution area and a high resolution area, to produce an area devoid of a valid pointer. In this case, an image on a layer that belongs to area 152f towards the lower resolution is enlarged to display an image included in area 152e. When the resolution of the displayed image increases to enter the 152s area, the index block designated by the pointer defined for the area is consulted to acquire a tiled image.
[0085] This allows an image other than the one displayed so far to be displayed once the resolution of a displayed image exceeds a certain level. A decorative touch can be added to the display, for example, by allowing a kana caption of a Chinese character to appear in the newspaper or by allowing an object to appear in the background of a photo as the resolution increases. Basically, a null pointer can be defined in the header for an arbitrary area, except that it is guaranteed that a valid pointer is defined for an area 152g at the top of the hierarchical structure, that is, the area that includes the image with the highest resolution. low. This ensures that the search through the hierarchical structure at least reaches area 152g and that a displayed image can be generated in this way, regardless of where, in other areas, null pointers are defined.
[0086] Figure 13 shows a method for defining, in the index block, how tiled images are shared in the presence of redundancy. Of the image areas that belong to the index block 160e, pointers that point to specific tile images are not defined for those areas that can be displayed using an image on another layer included in the index block 160e. Instead, null pointers are defined for these areas. In figure 13, pointers that point to specific tile images, that is, valid pointers, are defined for the shaded tile locations in index block 160e, and null pointers are defined for the blank areas.
[0087] To display an area of image 158 in the figure, an area 152h defined in header 150b and which includes the area of image 158 is identified first. Then, the index block 160e designated by the pointer defined for area 152h is acquired (arrow I). If, of the areas for which the index block 160e holds data, a null pointer is defined for an area found in an image in the second layer 162g and corresponding to the area of the image 158, the layers that belong to the same index block 160e are searched in the direction in which the image is reduced to identify a layer to which a valid pointer is defined for a reduced version of the image area 158.
[0088] In the illustrated example, first, the search arrives at an image in the first layer 162h (arrow J). However, a null pointer is also defined for the area corresponding to the image area 158. Therefore, the search continues until reaching the images in the 0 th layer 162i with the lowest resolution (arrow K). In images in the 0 th layer 162i, a valid pointer is defined in the area corresponding to the image area 158. Therefore, data for a tile image 170d designated by the pointer is acquired (arrow L). The displayed image of the 158 image area can be generated using the acquired data. Considering that the area of image 158 is the size of a tile image, a partial area 174 of tile image 170d, a reduced image, is enlarged as shown to render the area of image 158. Similar to the above, the location of the image area 174 corresponding to the area of image 158 can be easily computed by reference to the scale of area 174 in relation to the area of image 158.
[0089] By allowing the 160e index block to define a null pointer, it is not necessary to define pointers that point to tile images for all tile locations defined by the 160e index block. Consequently, data for tiled images can be compressed and the data volume for the index block can also be reduced. The relationship between images in a given index block remains unchanged in terms of how images are produced from one another by enlarging or reducing, even when a tile image is updated. Therefore, it is not necessary to update the pointers that point to all the tiled images so that the steps required for updating are minimized.
[0090] As in the header, valid pointers are defined for all tile locations in the images in the 0 th layer 162i, the layer with the lowest resolution in the index block. This ensures that, even if a null pointer is defined in another arbitrary area, the search through the layers in the reduction direction arrives at least in the images in the 0 th layer 162i, so that a displayed image is generated in this way. As an exception, of the areas defined by the header, the area at the top of the hierarchical structure, that is, the area that includes the layer with the lowest resolution, may inherently contain a relatively smaller number of tiled images, as exposed. In this case, all layers in the corresponding index block can include areas for which a null pointer is defined.
[0091] As exposed for the header, for a given area of the image in the index block, a valid pointer can be defined for the second layer in high resolution 162g and for the 0 th layer in low resolution 162i, and a null pointer can only be set for the first layer in medium resolution 162h. To display an area at the bottom left of the first layer in 162h medium resolution of figure 13, an enlarged version of the tile image designated by the pointer defined for the equivalent area in the 0 th layer of lower resolution 162i is used for display. If the resolution of the displayed image increases until the second layer 162g is used, the tile image designated by the pointer defined for the equivalent area in the second layer 162g is used to generate a displayed image. As in the case of the header, a display mode can be implemented in which a different image is presented for viewing when a given area is enlarged.
[0092] Thus, the use of the index block can define the sharing of a tiled image in a more detailed way (that is, in units of tiled images) than by using the header to define the sharing of a block of images. index. Figures 12 and 13 show that a null pointer is defined in the header or in the index block. Alternatively, null pointers can be defined in both. In this case, the sharing of an index block and a tiled image can be defined in similar steps. In the above example, a null pointer is defined in the header or in the index block to implement a modality in which a tile image is shared across different layers. To share a tile image on the same layer, pointers that point to the same index block can be defined for a plurality of areas, the areas in the hierarchical structure, which include the same layer. Alternatively, pointers that point to the same tile image can be defined for a plurality of tile locations in the index block.
[0093] Now, a description of the operation of the control unit 100b with the function of generating an image file will be given. Figure 14 is a flow chart showing steps according to which the control unit 100b shown in Figure 9 generates an image file. Initially, when the user or another processing module provides an input to designate image data, the image hierarchy generating unit 120 reads the designated image data from the hard drive 50 (S50). Then, the image hierarchy generation unit 120 generates hierarchical data that comprise image data of different resolutions by reducing the image data read, in the usual way, and by generating image data with a predetermined resolution (S52 ). Subsequently, the image splitting unit 122 divides the image in each layer into tiled images and stores the tiled images in main memory 60 (S54). In this process, the position of each tile image in the image and the address indicating the storage area in main memory 60 are mapped to each other, and the mapping is recorded. The recorded mapping is used to generate index blocks.
[0094] A file that already includes tiled images can be read in S50. In this case, the generation of hierarchical data in S52 and the division of the image in S54 can be omitted. By including a value that indicates a shift in the address of each tile image in relation to the starting address and the position in the image in the file, and by recording, in S54, the starting address in main memory 60 of the file read in S50, the relationship between the address of each tile image and the position in the image can be acquired.
[0095] Subsequently, the redundancy detection unit 124 examines the redundancy between the images, as described above (S58). More specifically, the redundancy detection unit 124 scans the images of the N-th layer to identify if there is any area of a commensurate size with the size of a tile image on the N + 1- th layer and which contains only a single color. . The step is repeated in the direction of higher hierarchies. In addition, the redundancy detection unit 124 examines the images to identify an area in the image of the N + 1-th layer that contains substantially the same amount of information as the image of the N-layer by examining a differential image produced between an image produced by enlarging the image of the N-th layer to the size of the N + 1-th layer and the image of the N + 1-th layer or by analyzing the frequency of the image of the N + 1-th layer. Any method ordinarily practiced in the field of image processing can be used for examination.
[0096] The header / index generation unit 126 creates a header and an index block in consideration of redundancy (S60). More specifically, if the entirety of an area defined in the header can be displayed by enlarging an image that belongs to an area of a lower resolution, a null pointer is set to the upper resolution area. For the other areas, corresponding index blocks are generated and pointers that point to the generated index blocks are defined in the header. Of the images in a plurality of layers that belong to each index block, a null pointer is defined for those image areas that can be displayed by enlarging a lower resolution image. For the other areas, pointers that point to tile images extracted from the areas are defined.
[0097] Subsequently, the unit for generating the image file 128 creates an image file that must ultimately be transmitted (S62). If the redundancy detection unit 124 does not detect redundancy, the image file generation unit 128 reads all data for tile images generated by the division in S54 from main memory 60. The image file generation unit 128 creates an image file that also contains the header and the index block generated in S60. If the redundancy detection unit 124 detects redundancy, the image file generation unit 128 reads only the data for the tiled images designated by the pointers defined in the index block generated in S60. The image file generation unit 128 creates an image file that also contains the header and the index block generated in S60.
[0098] A description of the operation of displaying an image using the image file thus generated will be given. Figure 15 is a flowchart that shows steps to display an image, using the image file according to the modality. The illustrated steps are implemented by the control unit 100a shown in figure 7. The flowchart primarily shows a process for reading necessary data from an image file in response to a request to change a display area, and for decoding and displaying the image in this way. The basic flow, which includes loading from a hard disk, pre-fetching the necessary data and writing to a temporary storage memory, is similar to the process illustrated in figure 5.
[0099] Initially, when the user enters a request to change an image displayed via the input device 20, while a part of an image is displayed on the display device 12, the input information acquisition unit 102 recognizes the request (S70). The tile image identification unit 110 derives the requested amount of change in the displayed image and again determines the coordinates of the frame to be displayed (S72). The amount of change in the displayed image represents the amount of movement in the vertical or horizontal direction in the virtual space and the amount of movement in the direction of depth. The coordinates of the frame that is to be displayed are determined by the coordinates of the frame in the display area that was displayed and the amount of change derived.
[00100] The tile image identification unit 110 consults the header and identifies an area to which the frame coordinates belong. The tile image identification unit 110 checks whether a valid pointer pointing to an index block is defined for the identified area (S74). If a valid pointer is defined (Y in S74), the tile image identification unit 110 acquires the index block designated by the pointer (S78). If a null pointer is defined (N in S74), the tile image identification unit 110 seeks the hierarchical structure in the direction in which the image is reduced to an area for which a valid pointer is defined. The tile image identification unit 110 acquires the index block pointed by the pointer defined for the detected area (S76, S78).
[00101] The tile image identification unit 110 fetches the index block to determine whether a valid pointer pointing to a tile image is defined for the area corresponding to the frame coordinates (S80). If a valid pointer is defined (Y in S80), the tile image identification unit 110 identifies information that uniquely defines the tile image pointed by the pointer, for example, the address and identification number of the tile image (S84 ). If a null pointer is defined (N in S80), the tile image identification unit 110 searches for the hierarchy in the index block in the direction in which the image is reduced. The tile image identification unit 110 searches for a layer in which a valid pointer is defined and identifies a tile image pointed by the defined pointer to the corresponding area on the layer thus identified (S82, S84).
[00102] If the buffer 70 does not store data produced by decoding the tile image (N in S86), the decoding unit 112 reads the data for the tile image from main memory 60 and decodes the data therein. way (S88, S90).
[00103] If an area for which a valid pointer is defined in the header is searched for in S76, or if a layer in the index block in which a valid pointer is defined is searched for in S82, the identified tile image must be enlarged and rendered. Therefore, when a tile image is identified through a step like this, the tile image identification unit 110 provides decoding unit 112 with information indicating the fact. The decoding unit 112 determines whether it is necessary to enlarge the tile image by reference to the information (S92).
[00104] If it is necessary to enlarge the image (Y in S92), the decoding unit 112 enlarges the tile image by reference to the frame coordinates acquired from the tile image identification unit 110 and stores necessary areas in the memory of temporary storage 70 (S94). The aforementioned process is similarly performed for the tile image already decoded (Y in S86). Regardless if the image is enlarged, the displayed image processing unit 114 renders, in the frame memory, an area in the tiled image that must be displayed again (S96). The rendering process includes a process of enlarging or reducing the data stored in the temporary storage memory 70, depending on the requested resolution.
[00105] Now, a description of the data structure of a pointer will be given in the header and in the index block. For example, given that hierarchical data comprising 11 layers is organized into index blocks, each comprising three layers comprising, respectively, 4 x 4, 8 x 8 and 16 x 16 tile locations, as shown in figure 8, the total number of areas defined by the header will be 4,162. A maximum of 336 tile locations are defined in a single index block. Since a pointer is defined for each of these elements, the data structure should desirably allow an efficient search for a pointer. A fixed-length array, associative array, or tree structure can be used as the data structure for a pointer.
[00106] If the pointer in a header is configured as a fixed-length arrangement, an identification number is assigned to each area. The identification number is used as an index to access the fixed-length pointer array. In this case, the identification of an area immediately results in the acquisition of a pointer, so that editing or searching can be carried out quickly. If an associative arrangement is used, an identification number is assigned to each area. The identification number is used as a key to access the associative array of pointers. In this case, only valid pointers need to be stored, so that the size of the header data is reduced.
[00107] If a tree structure is used, a pointer that connects areas is additionally defined. Inter-area pointers are sought, starting at the top of the hierarchical structure of the images, to identify a desired area. In this case, only those pointers near the top need to be updated when a layer is added at the top, as described below. Thus, different pointer data structures provide different advantages. An appropriate structure is selected, depending on the factor that weighs, such as memory cost or processing cost. A method ordinarily used in the search for data (for example, tree structure B) can be introduced as appropriate. Pointers in the index block are similarly configured. An appropriate method can be selected taking into account the processing speed, etc.
[00108] A pointer that points to an index block or a pointer that points to a tiled image can include the file name of a file that indicates a target index block or tiled image, or information about the location of a server connected to the network. This allows data for a single layer data set to include images from a plurality of files or web pages, or allows a single file to be shared by a plurality of layer data sets.
[00109] A description of the modification or revision of an image file comprising a header, an index block and a tile image, as shown, will be given. This can also be implemented by using the information processing system 1 shown in figure 1 and the information processing device 10 shown in figure 4 or a similar configuration. Figure 16 shows the configuration of the control unit 100c with the function of modifying an image according to this modality. The control unit 100c can also comprise the function of displaying images, like the control unit 100a of figure 7, or the function of generating an image file, like control 100b of figure 9, but the illustration is omitted in figure 16 Alternatively, the information processing device 10 provided with the control unit 100c with only the function shown in figure 16 can be provided separately from the image processing device provided with the functions of displaying images and generating an image file.
[00110] The control unit 100c comprises: an image file acquisition unit 318 configured to acquire an image file subject to modification; an update information acquisition unit 320 configured to acquire update information comprising an area to be updated by the modification and image data for the updated part; a tile image generating unit 322 configured to generate an updated tile image; a header / index block update unit 324 configured to update a header and an index block; a displayed image control unit 326 configured to display an image that is modified; and an image file generation unit 328 configured to generate a modified image file.
[00111] The image file acquisition unit 318 recognizes a user entry that designates an image subject to modification, reads the relevant image file from hard drive 50 and stores the read file in main memory 60. The image file includes data for a header, an index block and a tiled image, as described so far. The update information acquisition unit 320 acquires update information that the user enters via the input device 20, still viewing an image subject to modification displayed on the display device 12. As shown, the update information includes the area that must be updated and the post-update image data for this area. A specific example will be given below.
[00112] The tile image generation unit 322 identifies a tile image that needs to be updated when the post-update image is applied to the area that is to be updated, and generates a new tile image. If the area to be updated includes only a part of a given tile image, the original tile image is read from main memory 60 and a new tile image is generated by rewriting only the area that must be updated. If the entire tile image is included in the area to be updated, a new tile image is generated by extracting the post-update image. In this process, new tile images are generated for areas that must be updated in all layers that form the hierarchical data. Alternatively, only a specified layer can be updated depending on the user's request. This will produce a display mode in which a different image is displayed at a specified range of resolutions.
[00113] The tiled images thus generated are stored in main memory 60. In this process, the original image data stored by the image file acquisition unit 318 in main memory 60 are kept as is. The newly generated tile image is stored in another storage area. Subsequently, the header / index block update unit 324 rewrites the pointer defined in the index block stored in main memory 60 for the area that must be updated to point to the newly generated tile image. If an index block corresponding to the area is not found, a new index block is generated, and the null pointer in the header is replaced by a pointer that points to the generated index block.
[00114] The displayed image control unit 326 may comprise function blocks included in the control unit 100a shown in figure 7. Illustration is omitted in figure 16. The displayed image control unit 326 displays an image subject to modification in stages similar to those described in relation to figure 15. The header and the index block are updated as described above according to the modification of an image initiated by the user. In association with this, tile images used to render an image are updated. This makes it possible for the user to modify or revise an image, while still viewing the displayed image.
[00115] In this modality, a tile image generated again is added in a separate storage area without updating the data for the original tile image. The image is updated by changing the reference destination using the index block or the header. Compared to the case of searching the original tiled images to identify data for a tiled image subject to data modification and rewriting, the aforementioned method incurs a lower processing cost and is able to display the image that is modified more responsively. It is also easy to return to the original image in the middle of the modification.
[00116] The unit of generation of the image file 328 rewrites the data of the original tile image with the data of the tile image generated again, for example, when the user completes the modification. Alternatively, the newly generated tile image data is stored in a separate file from the original tile image data. The updated index block and the header are added to form an image file.
[00117] Now, a description of a specific example of data processing will be given when an image is modified or revised according to the scheme described above. Figure 17 shows an explanatory example of modifying an image. In relation to the figure, an ellipse and a triangle are drawn on an image 190 before modification. The image is considered to be modified as indicated by a modified image 192, in such a way that a figure of a star 194 is added at the top right of the image. Figure 18 shows schematically how pointers in the index block change in this modification. In relation to the figure, pointers are denoted by arrows. In order to avoid disorganized appearance of the drawing, only representative arrows are shown.
[00118] Prior to modification, pointers that point to respective index blocks 160 or null pointers are defined in the areas defined in header 150. For each tile location in each of a plurality of layers that belong to index block 160 designated by the pointer, pointers that point to respective tile images 170e generated before modification are defined. If the area at the top right of the pre-modification image 190 shown in figure 17 comprises a background of a single color, a single tile image can be shared in the area. This is indicated by a large number of broken line arrows that point to a single tile image 170f in figure 18. Naturally, pointers that point to individual tile images are defined for the other areas.
[00119] If a figure of a star 194 is added in the background part, as indicated by the modified image 192 in figure 17, new 170g tile images are created for corresponding areas in the respective layers, showing figures of a star in the respective resolutions. The 170g tile images are stored in a different storage than the initial 170e tile images. The pointers in index block 160 for the tile locations included in the area to be updated are updated to point to respective images in the new 170g tile images. Figure 18 shows post-update pointers that use mixed line arrows. After modification, the area subject to modification is no longer the background, so that pointers that point to the respective tile images are defined for the respective tile locations. Different areas of the area subject to modification may remain pointing to the initial tile images 170e.
[00120] Now, a description of steps that add a layer of a higher resolution will be given when an image like the figure of a star 194 of figure 17 added by the modification has a higher resolution than the image with the highest resolution in original hierarchical data. Figure 19 shows steps for adding a higher resolution layer. In relation to the figure, the pre-modification image is considered to comprise hierarchical data 200 comprising a 0 th layer 154i, a first layer 154j, a second layer 154k and a third layer 154l. Valid pointers pointing to index blocks are considered to be defined for all areas in the header. Areas with valid pointers are indicated by shading in the figure.
[00121] If the image is modified by adding a high resolution image at the top right, as shown in figure 17, tile images for areas subject to update are generated again for the respective layers, as shown. In this case, since the added image has a higher resolution than the third layer 154l with the highest resolution in the original hierarchical data, a new layer must be added. In the figure, the added layer is indicated as a fourth layer 154m. Areas for which tiled images must be generated again are indicated in black in the figure.
[00122] In this example, modification of the image using a high resolution image occurs only locally. Therefore, to display areas other than those subject to updating at the fourth layer 154m resolution, images at the third layer 154l, which have the highest resolution in the original hierarchical data, are enlarged for use in the display. For this purpose, null pointers are defined for areas on the fourth layer 154m different from the areas subject to update to place tile images on the third layer 154l for use. Areas for which null pointers are defined are indicated in white in the figure.
[00123] As described above, the sharing of a tile image can be defined by the use of a header or the use of an index block. Since the number of layers organized by a single index block is fixed, an existing index block may already have defined a layer corresponding to the fourth layer 154m that must be added. Because the fourth layer 254m is not found in the original hierarchical data, null pointers are defined for layer 154m in the index block.
[00124] In this case, the null pointers defined in the existing index block for the areas subject to update in the fourth layer 154m are updated in pointers that point to the tile images generated again. Null pointers for areas other than areas subject to update must remain unchanged. Pointers in the index block are also updated in the lower resolution layers.
[00125] Figure 20 illustrates how the header and the index block change when there were no index blocks to which the layer corresponding to the fourth layer 154m belongs before the addition. An area 152i in header 150 represents an area defined for hierarchical data prior to modification. Header 150 can define a plurality of areas, of course, but the plurality of areas is denoted by a single area 152i in the illustration. The 0 th layer 154i, the first layer 154j, the second layer 154k and the third layer 154l existing before the modification belong to area 152i. Pointers that point to the 160f index blocks are defined for the individual areas. Pointers pointing to existing tile images 170h are defined for the respective tile locations defined in the 160f index blocks (arrows in broken line).
[00126] To add the fourth layer 154m, a new area that includes the layer is added in header 150. As shown in figure 20, a valid pointer is defined only for an area 152j that includes the area subject to update, and pointers Nulls are defined for the other areas. In addition, a new 160g index block is generated, and a pointer that points to the 160g index block is defined for area 152j. If the fourth layer 154m is the only layer that is added, pointers that point to tiled images need to be defined only for one of the three layers that belong to the newly generated 160g index block that has the lowest resolution. Null pointers are defined for all areas on the remaining layers.
[00127] Pointers that point to respective tile images generated again 170i are defined for tile locations in the newly generated index block 160g that are included in the area subject to update (arrow in mixed line). The existing index block 160f is similarly updated in such a way that pointers defined for tile locations included in the area subject to update point to respective tile images generated again 170i. When an image file is generated, ultimately, the newly generated index block 160g can be appended at the end of the original image file or provided in a separate file. This allows pre-update and post-update images to be displayed.
[00128] Now, you will be given a description of the steps to increase the size of an image by adding a new area, such as a background or a rendered area, to an existing image. Figure 21 shows steps for adding a new area to an existing image. With respect to the figure, the image pre-addition is considered to comprise hierarchical data 204 comprising a 0 th layer 154i, a first layer 154j, a second layer 154k and a third layer 154l. A valid pointer that points to an index block is defined in a header for the area to which the layers belong.
[00129] To add an area to this image, 206 hierarchical data are generated in which new areas are added to the images of the respective layers in sizes corresponding to the respective resolutions. In the example of figure 21, hierarchical data 206 comprises a 0th layer 154n, a first layer 154o, a second layer 154p, a third layer 154q and a fourth layer 154r. The fourth layer 154r and the third layer 154q can be generated by adding tile images for additional areas indicated by shading in the tile images of the original image indicated in white. The second layer 154p is produced, depending on its size, by adding tile images indicated by shading, in which additional areas and original images are coexisting in the areas indicated in white, in which original tile images can be used unmodified. In the first layer 154o and the 0 th layer 154n, original images and additional areas are coexistent in all the tile images that form the layers. Therefore, the layers are indicated by shading.
[00130] As described above, depending on the layer, original tile images can be used unmodified when new areas are added to the image. The property can take advantage in such a way that existing index blocks are used unmodified, and new index blocks are generated only for areas (for example, additional areas) for which such blocks are needed. Figure 22 shows how the header and index blocks change when a new area is added. The white triangle in the header 150 of the figure indicates an area 208 where original tile images can be used. Shading indicates an area of change 210 added or modified.
[00131] The broken lines indicate the 0 th layer 154n, the first layer 154c, the second layer 154p, the third layer 154q and the fourth layer 154r of figure 21. In the respective layers, areas where original tile images can be used, areas to be added, and areas to be updated by addition are defined, resulting in the structure shown in figure 22. The areas defined by the header are generated from the top to the bottom of the hierarchical structure. Therefore, as header 150 changes, as explained, change also occurs in the partitions of the areas.
[00132] By designing the appropriate partitioning of the areas, as described below, the areas defined before the change can be reserved for use as much as possible, so that existing index blocks can be used unmodified, even if they occur a change in which the number of layers increases at the top of the hierarchical data, as shown in figure 22. Before a new area is added, pointers that point to index blocks 160h in figure 22 are defined for individual areas (not shown) in area 208. Pointers that point to the existing tile images 170h are defined for the tile locations defined in the 160h index blocks (arrows in broken line).
[00133] After the area is added, new blocks of index 160i corresponding to the added area are generated. Pointers that point to index blocks 160i are defined for individual areas in the new area 210 in header 150. Header 150 can define a plurality of individual areas (not shown) in area 210. Pointers (arrows in mixed line) that point to the respective new 170k tile images required as a result of adding the area are defined for the respective tile locations in the 160i index blocks so generated (arrows in mixed line). The new 170k tile images are area tile images or added tile images in which the added area and original image coexist, which are indicated by the shading in figure 21. If the added area represents a single color background , a single tiled image can be shared.
[00134] Since the area partition remains unchanged, the areas initially defined in the header can use the already defined pointers and, ultimately, the index blocks designated by these pointers. Therefore, there is no need to update pointers defined in the index blocks and pointing to tiled images. In the example above, the added area is located on the right and below the existing image. Whichever direction the image is expanded, change in the header and index blocks is minimized by dividing the image into areas, so that existing index blocks can continue to be used after expansion.
[00135] Figure 23 shows how the division into areas is defined in the header when a layer is added at the top of the hierarchical structure (for example, when the image size increases), as illustrated in figures 21 and 22. The hierarchical data initials 230 are partitioned into areas, such that the integral of the 0th until the first layer is defined as a first area 152k, the integral of the second to fourth layers is defined as a second area 152l, the areas produced by the division of the fifth through seventh layers according to the location in the image are defined as a third area 152, a fourth area 152n, etc. The structure of the areas is the same as the structure shown in figure 8.
[00136] As described above, it is guaranteed that the number of tile images that can be defined in a single index block is uniform, regardless of the index block. Since the first 152k area comprises only the 0 th layer comprising a single tile image and the first layer comprising 2 x 2 tile images, an index block 160j corresponding to the 152k area uses only the commensurate number of tile. Null (invalid) pointers are defined for the other tile locations.
[00137] In the index blocks shown in figure 23, the number of tile locations valid in the horizontal direction is indicated by rectangles in each layer. Thus, a rectangle and two rectangles are shown in the index block 160j corresponding to the first area 152k. All 4 x 4, 8 x 8 and 16 x 16 tile locations in the index blocks, such as 160k, 160l, etc., corresponding to the other areas (for example, a second area 152l, a third area 152m, etc. ) may become valid. Therefore, 4, 8 and 16 rectangles are shown.
[00138] Hierarchical data 232 results from the addition of a layer on top of the hierarchical data 230. The layers belonging to the first area before the addition must remain unchanged, and the added layer is additionally included in a first area 152o. The index block 160j corresponding to the first initial area 152k uses only two of the three layers made available as data areas. Therefore, the first area 152o can individually accommodate the addition of the layer by using the layer that used to be invalid. Therefore, the second to fourth layers included in the second area 152l in the original hierarchical data 230 (third to fifth layers after the layer is added) may remain in the same area. However, the images in the respective layers are expanded so that a new area is defined when the image is expanded and an index block is generated in this way. The same is true for layer below.
[00139] Hierarchical data 234 results from adding a layer on top of hierarchical data 232. The layers belonging to the first area before the addition must remain unchanged, and the added layer is additionally included in a first area 152p. This results in four layers included only in the first 152p area. Since the number of tile images made available in a corresponding index block 160m is greater than the number of tile images in the first area 152p, the 160m index block can accommodate information about the tile images for the four layers. This means that the layers below may belong to the same areas to which they belong in the initial 230 hierarchical data.
[00140] Hierarchical data 236 result from the addition of a layer in the hierarchical data 234. The five layers resulting from the addition of the additional layer in the four layers belonging to the first area 152p in the hierarchical data 234 are partitioned into two areas to produce a first area 152q and a second area 152r. By allowing the first area 152q to include the 0 th to the first layers and by allowing the second area 152r to include the second to fourth layers, an index block 160n corresponding to the first area 152q comprises only one layer comprising a a single tile image and a layer comprising 2 x 2 tile images, such as the index block 160j corresponding to the first area 152k in the initial hierarchical data 230. The second area 152r is similar to the second area 152l of the hierarchical data 230.
[00141] The layers below may belong to the same areas to which they belong in the initial 230 hierarchical data. By repeating the above changes, areas can be defined so as not to affect the other areas as much as possible when a layer is added at the top of the data hierarchical and to consequently use existing unmodified index blocks as much as possible.
[00142] When the image is expanded to add areas, as shown in figure 21, and when the tiled images are managed based on the sequence in the direction of the rasterization, as ordinarily done in image processing, tiled images of the areas added are inserted between the existing tiled images, requiring the rearrangement of tiled images and the reassignment of identification numbers. The greater the number of tiled images, the heavier the load required for processing. According to this modality, the image can be expanded to add more areas by making only a local change in the data, as shown. This results in low processing costs and responsive display of the data being updated and responsive generation of a fundamental image file.
[00143] A description of the operation implemented by the aforementioned configuration will be given. Figure 24 is a flowchart that shows steps taken by the image processing device when the user modifies or revises an image. Initially, when the user designates an image file subject to modification, the image file acquisition unit 318 reads the relevant image file from hard drive 50 and stores the file in main memory 60 (S110). This causes the displayed image control unit 326 to display the image defined by the image file on the display device 12 (S112).
[00144] When the user enters update information, still viewing the image displayed on the display device 12 (Y in S114), the update information acquisition unit 320 recognizes the information. The tile image generating unit 322 generates a new tile image for the updated part, and stores the generated tile image in main memory 60 (S116). The user can enter update information, for example, by dragging an image to be pasted, which is displayed in the vicinity of the image displayed on the display device 12 in the direction of a desired area using a pointing device. Alternatively, the user can directly draw a character or figure on the image using a pointing device or can enter a command to add a new region.
[00145] Subsequently, the header / index block update unit 324 updates the header and the index block, as appropriate, according to how the image is updated (S118). This causes the displayed image control unit 326 to update the display on the display device 12 by reading the data for the tile image consulted again and decoding the read data (S120). Steps S114 through S120 are repeated until the modification is completed and update information is no longer available.
[00146] When the completion of the modification is detected from user input, etc. (N in S114), the image file generation unit 328 reconstructs the image data by incorporating the newly generated tile image stored in main memory 60 in the image data in the original image file (S122). Alternatively, only the newly generated tile image is stored in a separate file. The image data thus reconstructed is transmitted as an image file together with the header and the index block updated in S118 (S124).
[00147] During the modification process, the header, the index block and the tiled image are stored in a storage area different from that of the existing data. When the user initiates an operation to cancel the update immediately preceding an image (Undo), the image can easily return to its initial state by restoring the reference destination of the data. Redoing an operation can also be performed only by changing the reference destination.
[00148] According to the aforementioned modality, the hierarchical data comprise three sets of data that include the header, the index block and the tile image. This allows a tiled image to be independent of its position in the image and facilitates management in the event of modifying an image. Additionally, since a tiled image can be shared by a plurality of areas in consideration of redundancy, the size of the tiled image data can be efficiently reduced.
[00149] When a tile image is shared by a plurality of areas, a null pointer is defined in the header or in the index block. In the area where the null pointer is defined, layers are searched for in the direction of a lower resolution to identify an area where a valid pointer is defined. This reduces the size of the data used for the index, compared to the case where pointers pointing to the respective tile images are defined in all areas.
[00150] Additionally, the header is used to divide the hierarchical structure into areas. An index block is mapped to each area. In this process, it is guaranteed that an area includes a plurality of layers. This allows for the organization of data that is likely to be displayed at once or subject to modification at once as a block. Inefficiency in using a storage area or processing (for example, loading unnecessary data into main memory) can be avoided more easily than in other circumstances. By ensuring that the fixed number of tiled images can be defined in an index block, fragmentation is prevented from occurring when data is cached and the management of a storage area is facilitated. Even if the image grows to a large size of the data, data that must be displayed or subject to modification can be accessed efficiently by dividing the hierarchical data into areas.
[00151] In the case of hierarchical data comprising 80,000 tile images, 200 index blocks are generated if the number of tile locations defined in an index block is 400. Sequential search to identify a given tile image will be required if the hierarchical data comprise merely a list of 80,000 tiled images. By dividing hierarchical data into areas, as in this modality, searching for a single area will be equivalent to searching 400 tile images, so that efficient access is possible.
[00152] Similarly, the number of index blocks required for images displayed at once on an ordinary image display device will be 1-2 on average. It is rare that 4 or more index blocks are needed. For this reason, the division into areas allows for a smaller size of the required index data at once and saves the main memory more efficiently than when indexes that point to tiled images are kept without division into areas. As a result, this increases the likelihood that a plurality of hierarchical data sets can be processed simultaneously without overloading main memory.
[00153] Pointers defined in the header or in the index block can define files that store index blocks or tiled images. This allows a single set of hierarchical data to use data read from a plurality of index block files or a plurality of tiled image files, enabling more flexible generation or updating of hierarchical data. Different files do not need to be stored on the same recording medium. Therefore, massive image data that exceeds, for example, 4 GB can be generated. This allows uncompressed images, high dynamic range (HDR) images, attribute information, etc. processed.
[00154] Similarly, a plurality of hierarchical data sets can share a data file that is used. For example, the data sizes of individual hierarchical data sets can be reduced by sharing the image data that can be reused as a template (for example, a magazine header or footer) instead of keeping the data in the sets individual. Not only tiled images, but also data for index blocks can be shared.
[00155] This can take advantage to facilitate updating content that uses hierarchical data. For example, you can distribute a file that includes data for a header, an index block and a tiled image as the content of a regular version. A file containing tiled images of the higher resolution layers is provided separately as an add-on. The header and index block of the regular version are configured to contain pointers that point to complementary tiled images, which will not be displayed unless these tiled images are obtained because the pointers remain invalid. When the user updates the content, the complementary file is transferred so that the pointers to the tile images stored in the file become valid. In this way, the complementary images are displayed. This will make the update easy without requiring changes to the content file of the regular version. The user may be charged for the update. Second Mode
[00156] In the first modality, an image file is generated and modified, primarily by reference to the information entered by the user. In this modality, a description will be given of a way in which a graphics processor renders hierarchical data. By introducing hierarchical data in the rendering, only those steps for a required resolution need to be performed, as described above in the first modality. This allows large, high-resolution images to be processed efficiently. This modality can be implemented by the information processing device 10 shown in figure 4 of the first modality. Figure 25 shows with further details the structure of the control unit 100 and the main memory 60 of the information processing device 10 according to the modality.
[00157] The control unit 100 includes a main processor 400 and a graphics processor 402. The main processor mainly executes an application program executed on the information processing device 10 and controls the other functional blocks included in the control device. information processing 10. Graphics processor 402 performs image processing according to a request for image processing from main processor 400. Data required for image processing are stored in main memory 60. Graphics processor 402 reads the data to perform image processing in this way and writes the result to main memory 60, etc.
[00158] Graphics processor 402 uses model data 418 stored in main memory 60 to render data according to a request for image processing from main processor 400. Graphics processor 402 updates a temporary storage of color 414 and a temporary storage Z 416 stored in main memory 60. In this embodiment, both the color temporary storage 414 and the temporary storage Z 416 are of a hierarchical data structure, as shown in figure 25. In other words, the temporary stores have a structure in which a pixel plane of the color value is hierarchical, which stores color information of the pixels of an image to be rendered, and a pixel plane of the Z value, which stores information about the depth of the pixels to be rendered. from a viewing point, the plans being hierarchized according to a plurality of resolutions of the image to be rendered gives. The hierarchical data may comprise a header, index blocks and tile data, as described in the first embodiment. The tile data represents data for tile locations defined by dividing an image plane into each layer into predetermined sizes.
[00159] The graphics processor 402 comprises a rendering unit 404 configured to render according to a request from the main processor 400 and a hierarchical data update unit 406 configured to reflect the result of rendering by the rendering unit 404 in the respective layers in the color buffer 414 and the buffer buffer Z 416. The graphics processor 402 additionally comprises storage areas temporarily used for rendering, that is, a GPU 408 color buffer, a GPU 410 Z buffer, and a dirty mask 412. Dirty mask 412 has an area the same size as the GPU 408 color buffer and the GPU 410 Z buffer, each pixel storing 1-bit information. When GPU 408 color buffer and GPU 410 buffer Z are updated by rendering, the updated pixel value changes to allow pixel detection.
[00160] Now, a description of the operation performed for rendering by the image processing device with the aforementioned structure will be given. Figure 26 is a flowchart showing steps for rendering hierarchical data. Initially, main processor 400 requests that graphics processor 402 perform rendering (S200). As shown, data required for rendering is stored in main memory 60. In addition to model data 418, main memory 60 also stores color temporary storage 414 and temporary storage Z 416 with an exposed hierarchical structure.
[00161] Rendering unit 404 of graphics processor 402 identifies a layer and area in color temporary storage 414 and temporary storage Z 416 subject to rendering, based on model data 418, etc. (S202). Subsequently, the rendering unit 404 reads the data for the identified area from the 414 color buffer and the Z 416 buffer and populates the GPU buffer, comprising the 408 GPU buffer and the Z buffer GPU 410, with the data read (S204). As in the method of identifying a tile image described in the first modality, the header and the index block are searched in the declared order to identify tile data in the layer and area subject to rendering. Therefore, the data is read in tile data units.
[00162] In this process, the rendering unit 404 initializes the dirty mask 412 (S206). For example, the rendering unit 404 sets the pixel values to "0". The 404 rendering unit follows the generally known method of rendering using model 418 data. The 404 rendering unit updates the GPU 408 color buffer and the GPU 410 Z buffer as appropriate and updates the pixel values subject to update in dirty mask 412 at "1" (S208).
[00163] The rendering unit 404 updates the hierarchical data in the color temporary storage 414 and in the temporary storage Z 416 in main memory 60, based on the rendering result in the color temporary storage of the GPU 408 and in the temporary Z storage of the GPU 410 (S210). More specifically, the rendering unit 404 searches for the layer and area in the hierarchical data subject to rendering to identify a tile location that includes pixels subject to update by reference to the pixel values in the dirty mask 412. The rendering unit 404 updates the data for the tile location based on the rendering result in the GPU 408 color temporary storage and the GPU 410 Z temporary storage.
[00164] In this process, the 404 rendering unit generates new tile data and updates the pointer defined in the index block on a pointer that points to the new tile data. If frequent updating is expected, the generation of tile data can be omitted and only the pointer can be updated. Tile data can be generated only when the image is finished. The 404 rendering unit generates new tile data for areas that belong to different layers of the layer subject to rendering and corresponding to the updated area, and updates the pointers defined in the index blocks on pointers that point to the new tile data. Different layers have different resolutions, so that the rendering result is enlarged or reduced to be reflected in the new tile data.
[00165] If the tile data for the area subject to rendering is not found in the color temporary storage 414 and the temporary storage Z 416 in the first location (for example, if an image of higher resolution than that existing hierarchical data is rendered), the corresponding area in the above layer is enlarged to fill the GPU 408 color buffer and the GPU 410 Z buffer with the enlarged data. After rendering, only the updated tile data is reflected in the color temporary storage 414 and the Z 416 temporary storage, and the pointer in the header or index block is updated. Subsequently, tile data is generated for the different layers of the layer subject to rendering, and pointers are updated in this way, similarly to the exposed. The process is similar to the process described in the first modality in relation to figure 19.
[00166] If, due to the enlargement of the rendering result, in order to reflect the rendering result in the hierarchical data in a layer below the layer subject to rendering, in S210, the enlarged rendering result includes some tile location in the layer below, the included tile data is removed and the pointer that points to the tile data becomes invalid. This can reduce the size of the tile data. To display the image, the layer in which the pointer is valid is identified by a search, so that the image in the layer above is enlarged, as described in the first modality.
[00167] Color temporary storage 414 and temporary storage Z 416 can store a header and index blocks individually or share a header and index blocks. If the header and the index blocks are shared, a given index block defines a plurality of pointers that point to a single tile location. In other words, a pointer that points to the tile data in the color buffer and a pointer that points to the tile data in the buffer Z are defined. This can compress the data for the header and index blocks rather than for individual storage of the header and index blocks. When an image is finished after rendering, pointers in the index blocks that point to the tile data in temporary Z storage that are no longer needed can be deleted. The plurality of pointers defined in an index block for a single tile location may not point to color temporary storage or Z temporary storage, depending on the image content.
[00168] A plurality of color temporary stores can share a Z temporary store. For example, two sets of data can become available for color temporary stores and one data set can become available for a Z temporary store. Rendering can be performed by the use of a pair comprising a color buffer and the Z buffer and a pair comprising the other color buffer and the Z buffer. In this case, it is more efficient for the buffer to maintain the header and the index blocks individually for the purpose of control.
[00169] The aforementioned modality that renders the color temporary storage and the Z temporary storage as hierarchical data can be applied for anti-aliasing. In other words, hierarchical data that comprises a higher resolution layer and a lower resolution layer that should be used for display is generated. In ordinary oversampling, a storage area for a higher resolution image needs to be reserved even for an area devoid of any object (for example, a single color background). Meanwhile, by configuring these temporary stores as hierarchical data, as shown, an area that will be produced by enlarging a lower resolution layer does not need to be rendered in high resolution.
[00170] By allowing the user to designate a required resolution area by area, only a minimum number of areas in each layer needs to be rendered, so that the efficiency of memory usage increases. For frequent updating of tile data and responsive image display, generated tile data does not need to be compressed. Different anti-aliasing methods can be employed depending on the relative size of the target area. For example, oversampling can be performed if anti-aliasing is required in full of an image.
[00171] Anti-aliasing can be accomplished by using hierarchical data in one of two approaches. In one approach, only the image in the lowest layer is rendered and scaled-down versions are generated once the image is finalized. In another approach, rendering is performed on the layer designated by the user and the corresponding areas on all layers are concurrently updated. The first approach helps to reduce the frequency of reduction processes. The second approach facilitates the forecast of the cost required for pixel generation, such as rasterization, shading process, rendering output channel (ROP), pixel transfer, etc. However, the second approach requires frequent transfer of original images, enlarged images, reduced images, etc. between graphics processor 402 and main memory 60, so that the transfer bandwidth tends to be heavily overloaded.
[00172] In general, not only anti-aliasing, but also rendering hierarchical data requires frequent scaling (ie, enlargement or reduction) of images in order to reflect the result of rendering in the layers. When image data is transferred between the graphics processor and main memory each time a scaling process is performed, the transfer bandwidth between the processor and memory tends to be heavily overloaded. To address this problem, a scheduler 422 responsible for scheduling can be built into main memory 60, as shown in figure 27.
[00173] With this, a graphics processor 420 and main memory 60 need only change the image of the resolution subject to rendering. The scheduler 422 ensures that the rendering result is reflected in the layers in the hierarchical data in main memory 60. As a result, the transfer bandwidth between graphics processor 420 and main memory 60 is prevented from being heavily overloaded, so that hierarchical data can be efficiently rendered. Scheduler 422 may not be built into main memory 60 and can be connected to main memory 60 using a bus separate from the bus between graphics processor 420 and main memory 60. This configuration can show equal effectiveness.
[00174] The aforementioned modality can be implemented using an ordinary graphics processor. An ordinary graphics processor employs an approach, whereby memory addresses close to each other are assigned to a neighborhood in the image in order to reduce the frequency of read and write operations in main memory when addressing a store internal frame, that is, the aforementioned GPU temporary storage. There is a restriction in which areas used in the GPU's temporary storage need to occupy a continuous address space. In other words, the GPU's temporary storage cannot be filled with an area with discontinuous addresses. Figures 28 and 29 illustrate a process that addresses the constraint.
[00175] In relation to figure 28, it is considered that the addresses "11" - "18" are assigned when an area 430 that is a part of the image in the third layer in the hierarchical data is rendered. In this case, the graphics processor 402 is able to fill the temporary storage of the GPU with area 430. It is considered that an area 432 that is a part of the third layer is subsequently rendered, as shown in figure 29. The area 432 includes areas to which addresses "11" and "12" are already assigned in figure 28. In this case, addresses "19" - "24" are assigned to the other areas in area 432. Such an address assignment results in area 432 including an area where addresses are discontinuous, with the result that the graphics processor 402 cannot fill the GPU's temporary storage with area 432.
[00176] Therefore, area 432 needs to be temporarily copied to a separate storage area and new continuous addresses (addresses "50" - "57" in figure 29) are assigned to area 432 before filling the GPU temporary storage with the area 432. To reduce the associated processing cost, a table that maps the positions in the area to the assigned addresses can be provided in the memory controller (not shown) in main memory 60 according to this modality. In the case of area 432 of figure 9, the positions in the area are indicated by "0" - "7" in the order of the rasterization. Mapping is established in such a way that "0" = "19", "1" = "20", "2" = "21", "3" = "22", "4" = "23", "5" = "24", "6" = "11" and "7" = "12".
[00177] By translating addresses in main memory 60 in this way, an area to which discontinuous addresses are assigned can be rendered without requiring a copy for the purpose of continuous filling, as shown in figure 29. By the construction of scheduler 422 in memory 60 , as shown in figure 27, most steps to generate and update hierarchical data, including the generation of enlarged or reduced images, can be performed exclusively in main memory 60.
[00178] The rendering of hierarchical data as described in this modality provides the advantages that include random access to tile data using a header and index blocks, concurrent processing of a plurality of hierarchical data sets enabled by the division of a header and cost resulting reduced memory for each hierarchical data set, and the ability to add, delete and update tile data in the rendering process. These features are the same as the texture requirements, that is, random access, multitexture and rendering target. Therefore, hierarchical data can be treated in the same way as texture data, so that hierarchical data can be placed in extensive applications in which computer graphics are used.
[00179] An explanation based on an exemplary modality is described above. It is intended that the modality is only illustrative, and it will be obvious to those skilled in the art that various modifications to the constituent elements and processes can be developed, and that such modifications are also within the scope of the present invention. DESCRIPTION OF REFERENCE NUMBERS
[00180] 1 information processing system, 10 information processing device, 12 display device, 20 input device, 38 tile image, 50 hard disk drive, 60 main memory, 100 control unit, 102 drive unit input information acquisition, 106 loading block determination unit, 108 loading unit, 110 tile image identification unit, 112 decoding unit, 114 displayed image processing unit, 120 image hierarchy generating unit, 122 image splitting unit, 124 redundancy detection unit, 126 header / index block generation unit, 150 header, 160 index block, 170 tiled image, 318 image file acquisition unit, 320 image unit acquisition of update information, 322 tile imaging unit, 324 header / index block update unit, 326 displayed image control unit, 328 u image file generation unit, 400 main processor, 402 graphics processor, 404 rendering unit, 406 hierarchical data update unit, 408 GPU color buffer, 410 GPU Z buffer, 412 dirty mask, 414 color temporary storage, 416 Z temporary storage, 418 model data, 422 scheduler, 520 graphics processor INDUSTRIAL APPLICABILITY
[00181] As described above, the present invention can be applied to information processing devices, such as computers, gaming devices, image processing devices, image rendering devices, etc.

权利要求:
Claims (27)
[0001]
1. Image file generation device, characterized by the fact that it comprises: an image hierarchy generation unit (120) configured to generate a plurality of parts of image data with different resolutions from input image data and to generate hierarchical data formed by hierarchizing the plurality of pieces of data according to the resolution; an image splitting unit (122) configured to divide image data into each layer of hierarchical data into tile images of a predetermined size; an index creation unit (126) configured to create index data that map tile locations defined by dividing an image into each layer of hierarchical data in tile images to data for the tile images used to render the tile locations ; and an image file generation unit (128) configured to generate an image file that includes data for the tiled images and the index data, where the index block creation unit creates the index data in the form of index blocks (160) that organize index data for respective tile locations, for each of the areas produced by the division of a hierarchical structure that comprises the hierarchical data in a virtual space defined by an image plane of the hierarchical data and a geometric axis of resolution perpendicular to the image plane.
[0002]
2. Image file generation device according to claim 1, characterized by the fact that the index block creation unit additionally generates header data (150) that map areas produced by dividing the hierarchical structure into index blocks (160) corresponding, and the image file generation unit (128) includes the header data (150) and the index blocks (160) in the index data.
[0003]
Image file generation device according to either of claims 1 or 2, characterized by the fact that it additionally comprises: a redundancy detection unit (124) configured to analyze an image in a layer of a given resolution for identify an area in a higher resolution image on a layer with a higher resolution that can be rendered by using data for a tile image included in the image in the given resolution layer, where the index creation unit (126 ) maps, in the index data, data for a tile image only up to different areas of the area in the higher resolution image identified by the redundancy detection unit (124).
[0004]
An image file generation device according to either of claims 1 or 2, characterized by the fact that it additionally comprises: a redundancy detection unit (124) configured to analyze an image in a layer of a given resolution for identify an area in a higher resolution image on a layer with a higher resolution that can be rendered by using data for a tile image included in the image in the given resolution layer, where, when the area in the resolution image The upper unit identified by the redundancy detection unit (124) extends over an entire area produced by dividing the hierarchical structure, the image file generation unit (128) excludes the corresponding index block (160) from the image file. to that area in the higher resolution image.
[0005]
An image file generation device according to any one of claims 1 to 4, characterized in that the index data maps data to a single tile image to a plurality of tile locations, and the image file does not include data for a tile image (170) not mapped in the index data.
[0006]
6. Image file generation device according to any one of claims 1 to 5, characterized by the fact that each of the areas produced by the division of the hierarchical structure includes corresponding areas in the images in a plurality of layers included in the hierarchical data.
[0007]
7. Image file generation device according to any one of claims 1 to 6, characterized by the fact that the number of tile locations that can be mapped in a single index block to data for tile images is uniform across of all index blocks (160).
[0008]
8. Image processing device, characterized by the fact that it comprises: a storage device configured to store an image file that includes data for tile images produced by dividing, in a predetermined size, images into hierarchical data that comprise data images that represent a single image at different resolutions, and that includes index data that map tile locations defined by dividing an image into each layer of hierarchical data in tile images (170) to data for the tile images used for rendering the tile locations; an input information acquisition unit (102) configured to recognize a user request to change a display area, the user request including movement in an image plane or change of resolution; and a tile image identification unit (110) configured to identify data for a tile image necessary to render an area that must be displayed again according to the request to change a display area by reference to the index block; a display image processing unit configured to read the data for the identified tile image from the storage device and render the area that must be displayed again, where the index data is in the form of index blocks (160 ) that organize index data for respective tile locations, for each of the areas produced by the division of a hierarchical structure that comprises the hierarchical data in a virtual space defined by an image plane of the hierarchical data and a geometric axis of resolution perpendicular to the image plane.
[0009]
Image processing device according to claim 8, characterized in that the index data includes header data (150) that map areas produced by dividing the hierarchical structure up to the corresponding index blocks (160), tile image identification unit (110) identifies data for the required tile image by searching for header data (150) based on information about an area that must be displayed again, by identifying one of the areas produced by the division the hierarchical structure that includes the area to be displayed and the reference to the index block mapped to the identified area.
[0010]
10. Image processing device according to claims 8 or 9, characterized in that the index data maps invalid data to a tile location in an image that can be rendered by using data for an included tile image in an image on a lower resolution layer, when the invalid data is mapped in the index data to a tile location included in the area that is to be displayed again, the tile image identification unit (110) fetches the hierarchical data in a direction in which an image is scaled down to a tile location mapped to data for a specific tile image included in an area that is a scaled-down version of the area that must be displayed again, to identify data for the specific tile image, and the displayed image processing unit renders the area that must be displayed again by zooming in on the specific tile image identified by the id unit tile image identification (110).
[0011]
11. Image processing device according to claim 9, characterized by the fact that when a tile location in an image can be rendered by using data for a tile image included in an image in a layer of a lower resolution extends over the whole of an area produced by dividing the hierarchical structure, the header data (150) maps invalid data up to that area, when the invalid data is mapped in the header data (150) to an area that includes the area that must be displayed and produced again by dividing the hierarchical structure, the tile image identification unit (110) searches for hierarchical data in a direction in which an image is reduced to an area that includes a reduced version of the area that it must be redisplayed and mapped to a specific index block to identify the specific index block and identify a reduced version tile image in this way, and the displayed image processing unit renders the area that must be displayed again by enlarging the tiled image identified by the tiled image identification unit (110).
[0012]
Image processing device according to any one of claims 8 to 11, characterized in that the index data maps a tile location to the data for a tile image that belongs to a file other than the image file , designating file identification information, and the displayed image processing unit reads data into a tile image used in the area that must be displayed again from the file designated by the index data and renders an image in this way.
[0013]
13. Image processing device, characterized by the fact that it comprises: an image file acquisition unit (318) configured to read an image file that includes data for tile images produced by the division, in a predetermined size, of images in hierarchical data that comprise image data that represent a single image in different resolutions, and that includes index data that map tile locations defined by dividing an image into each layer of hierarchical data in tile images to data for the images in tiles used to render tile locations; an update information acquisition unit (320) configured to acquire update information on an image displayed by using the image file; a tile image generating unit (322) configured to generate data for a new tile image used to render an area to be updated based on the update information; an index data update unit (324) configured to update data for a tile image mapped to a tile location defined in the index data as included in the area to be updated in the data for the new tile image; and an image file generating unit (328) configured to generate an image file that includes data for the tiled image generated by the tiled image generating unit and the index data updated by the index data update unit , in which the index data is in the form of index blocks (160) that organize index data for respective tile locations, for each of the areas produced by dividing a hierarchical structure comprising the hierarchical data in a defined virtual space by an image plane of the hierarchical data and a geometric axis of resolution perpendicular to the image plane.
[0014]
14. Image processing device according to claim 13, characterized by the fact that the tile imaging unit generates data for new tile images used to render an area to be updated, for all the layers that make up the tiles. hierarchical data, and the index data update unit updates data for tile images mapped to tile locations included in the area to be updated in all layers.
[0015]
15. Image processing device according to claim 13, characterized by the fact that the tile imaging unit generates data for a new tile image used to render an area to be updated only for a specific layer designated by update information, and the index data update unit updates data for a tile image mapped to a tile location included in the area to be updated on the specific layer.
[0016]
16. Image processing device according to any one of claims 13 to 15, characterized by the fact that, when the update information designates recording of an image with a higher resolution than the highest resolution in the hierarchical data before the update , the index data update unit additionally defines tile locations in a new layer that has the resolution of the recorded image and that must be added to the hierarchical data, and maps those of the defined tile locations that must be updated to the images in tiles that include the recorded image and that are generated again by the tile image generating unit.
[0017]
17. Image processing device according to any one of claims 13 to 15, characterized in that, when the update information designates the addition of a new image area to an image before the update, the image update unit index data again generates an area corresponding to a part of the hierarchical structure that changes as a result of the addition of the new image area and additionally generates the index block corresponding to the generated area.
[0018]
18. Image processing device according to claim 17, characterized by the fact that the tile imaging unit identifies an area of the tile images in which the added image area and the image before the update coexist and generates data for tile images where coexistence occurs, in addition to data for a tile image of the added image area.
[0019]
19. Image file generation method, characterized by the fact that it comprises: generating a plurality of parts of image data with different resolutions from the input image data, generating hierarchical data formed by hierarchizing the plurality of parts of data according to the resolution and store the data in a memory; read image data in each layer of hierarchical data from memory and divide the read data into tile images of a predetermined size; create index data that map tile locations defined by dividing an image into each layer from hierarchical data in tile images to data for the tile images used to render the tile locations, and store the index data in memory; and generate an image file that includes data for the tiled images and the index data, and store the generated file in memory, where the creation of index data creates the index data in the form of index blocks (160) that organize index data for respective tile locations, for each of the areas produced by the division of a hierarchical structure that comprises the hierarchical data in a virtual space defined by an image plane of the hierarchical data and a geometric axis of resolution perpendicular to the plane of image.
[0020]
20. Image processing method, characterized by the fact that it comprises: reading, from a memory, an image file that includes data for tile images produced by dividing, in a predetermined size, images into hierarchical data that comprise image data that represents a single image at different resolutions, and that includes index data that map tile locations defined by dividing an image into each layer of hierarchical data in tile images to data for the tile images used to render the tiles. tile locations; recognize a user request to change a display area that includes movement in an image plane and to change the resolution; identify data for a tile image needed to render an area that must be displayed again according to the request to change a display area by reference to the index block; and read the data for the identified tile image from memory, render the area that must be displayed again and display the rendered data on a display device; where the index data is in the form of index blocks (160) that organize index data for respective tile locations, for each of the areas produced by dividing a hierarchical structure comprising the hierarchical data in a virtual space defined by an image plane of the hierarchical data and a geometric axis of resolution perpendicular to the image plane.
[0021]
21. Image processing method, characterized by the fact that it comprises: reading, from a memory, an image file that includes data for tile images produced by dividing, in a predetermined size, images into hierarchical data that comprise image data that represents a single image at different resolutions, and that includes index data that map tile locations defined by dividing an image into each layer of hierarchical data in tile images to data for the tile images used to render the tiles. tile locations; acquire information about the update on an image displayed by using the image file; generate data for a new tile image used to render an area to be updated, based on the update information, and store the data generated in memory; update data for a tile image mapped to a tile location defined in the index data as included in the area for the data for the new tile image; and generate an image file that includes data for the tile image generated by the tile imaging unit and the index data updated by the index data update unit, and store the data in memory, where the index are in the form of index blocks (160) that organize index data for respective tile locations, for each of the areas produced by the division of a hierarchical structure that comprises the hierarchical data in a virtual space defined by an image plane of the hierarchical data and a geometric resolution axis perpendicular to the image plane.
[0022]
22. Non-transient computer-readable recording media, characterized by the fact that it comprises computer-readable instructions that, when read by a computer, operate with the same operation as: a module configured to generate a plurality of pieces of image data with different resolutions from input image data, generate hierarchical data formed by hierarchizing the plurality of pieces of data according to the resolution and storing the data in a memory; a module configured to read image data in each layer of hierarchical data from memory and which divides the read data into tile images of a predetermined size; a module configured to create index data that map tile locations defined by dividing an image into each layer of hierarchical data in tile images to data for the tile images used to render the tile locations, and store the index data in the memory; and a module configured to generate an image file that includes data for the tiled images and the index data, and store the file in memory, where the module for creating index data creates the index data in the form of blocks of index (160) that organize index data for respective tile locations, for each of the areas produced by the division of a hierarchical structure that comprises the hierarchical data in a virtual space defined by an image plane of the hierarchical data and a geometric axis of resolution perpendicular to the image plane.
[0023]
23. Non-transient computer-readable recording media, characterized by the fact that it comprises computer-readable instructions that, when read by a computer, operate with the same operation as: a module configured to read, from a memory, a file image that includes data for tile images produced by the division, in a predetermined size, of images in hierarchical data that comprise image data that represent a single image in different resolutions, and that includes index data that map tile locations defined by splitting an image into each layer from hierarchical data into tiled images to data for the tiled images used to render the tile locations; a module configured to recognize a user request to change a display area that includes movement in an image plane and to change the resolution; a module configured to identify data for a tile image needed to render an area that must be displayed again according to the request to change a display area by reference to the index block; and a module configured to read the data for the tile image identified from memory, render the area that must be displayed again and display the rendered data on a display device; where the index data is in the form of index blocks (160) that organize index data for respective tile locations, for each of the areas produced by dividing a hierarchical structure comprising the hierarchical data in a virtual space defined by an image plane of the hierarchical data and a geometric axis of resolution perpendicular to the image plane.
[0024]
24. Non-transient computer-readable recording media, characterized by the fact that it comprises computer-readable instructions that, when read by a computer, operate with the same operation as: a module configured to read, from a memory, a file image that includes data for tile images produced by the division, in a predetermined size, of images in hierarchical data that comprise image data that represent a single image in different resolutions, and that includes index data that map tile locations defined by splitting an image into each layer from hierarchical data into tiled images to data for the tiled images used to render the tile locations; a module configured to acquire information about the update on an image displayed by using the image file; a module configured to generate data for a new tile image used to render an area to be updated, based on the update information, and to store the data generated in memory; a module configured to update data for a tile image mapped to a tile location defined in the index data as included in the area for the data for the new tile image; and a module configured to generate an image file that includes data for the tile image generated by the tile imaging unit and the index data updated by the index data update unit, and store the data in memory, in that the index data is in the form of index blocks (160) that organize index data for respective tile locations, for each of the areas produced by dividing a hierarchical structure comprising the hierarchical data in a virtual space defined by a image plane of the hierarchical data and a geometric axis of resolution perpendicular to the image plane.
[0025]
25. Data structure of an image file read from a storage device to display at least part of an image on a screen, characterized by the fact that the data structure maps data to tile images produced by the division, at a predetermined size, from image data representing a single image at different resolutions to index data, the index data mapping tile locations defined by dividing an image into each layer of hierarchical data in tile images to data for the tiled images used to render the tile locations, the hierarchical data being formed by the hierarchy of said image data according to the resolution, and where the index data is in the form of index blocks (160) that organize data from index for respective tile locations, for each of the areas produced by dividing a hierarchical structure that comprises hierarchical data in a vi space rtual defined by an image plane of the hierarchical data and a geometric axis of resolution perpendicular to the image plane.
[0026]
26. Data structure of an image file according to claim 25, characterized by the fact that the number of tile locations that can be defined in a single index block is uniform across all index blocks (160) .
[0027]
27. Data structure of an image file according to claim 25 or 26, characterized in that the index data maps invalid data to a tile location in an image that can be rendered by using data for an image tiles included in an image on a lower resolution layer, to indicate that an image can be rendered using the included tile image.

类似技术:

---

公开号 | 公开日 | 专利标题

---

BR112012009893B1|2021-02-02|devices and methods of image file generation and image processing, non-transient computer-readable recording media, and data structure of an image file

---

US8798397B2|2014-08-05|System and method for converting source image data to tile data

---

Goode et al.2013|OpenSlide: A vendor-neutral software foundation for digital pathology

---

EP2464093B1|2016-09-21|Image file generation device, image processing device, image file generation method, and image processing method

---

EP2330587B1|2019-04-10|Image processing device and image processing method

---

US9373308B2|2016-06-21|Multi-viewport display of multi-resolution hierarchical image

---

JP4852012B2|2012-01-11|Apparatus for processing image, method and program for detecting update of image

---

CN108664619B|2020-06-19|Primitive storage and scheduling method for mass line-drawing topographic map of tile-like technology

---

US10460419B2|2019-10-29|Graphics rendering using a hierarchical acceleration structure

---

JP2012027705A|2012-02-09|Image processing device, image display device, image processing method, and data structure of image file

---

US5986661A|1999-11-16|Graphics output system with bounded updating

---

CN107479871B|2020-09-18|Massive vector data access and rendering method, electronic device and storage medium

---

JP5368254B2|2013-12-18|Image file generation device, image processing device, image file generation method, image processing method, and data structure of image file

---

US10013474B2|2018-07-03|System and method for hierarchical synchronization of a dataset of image tiles

---

US7650345B2|2010-01-19|Entity lookup system

---

JP5296656B2|2013-09-25|Image processing apparatus and image processing method

---

CN102253841A|2011-11-23|Small-scale graphical user interface system

---

JPH09269884A|1997-10-14|Method for conducting function strengthening-type locator hit test of window and data processing system

---

FI117655B|2006-12-29|A method for processing a computer-aided polygon model, a device and a computer program

---

Xu2017|Construction of a Responsive Web Service for Smooth Rendering of Large SSC Dataset: and the Corresponding Preprocessor for Source Data

---

KR101656619B1|2016-09-09|RBI-based Subgraph Listing Method

---

Xu2017|Construction of Responsive Web Service for Smooth Rendering of Large SSC Dataset

---

Pavlovec et al.2021|Rapid Labels: Point-Feature Labeling on GPU

---

Samet et al.0|levels. The lowest level |

---

Samet et al.0|PROCESSING ct-: ocR. u> rnc DATA wrru QUADTREI-IS ‘

同族专利:

---

公开号 | 公开日

---

CN102667864B|2015-06-03|

---

BR112012009893A2|2017-02-21|

---

CN102667864A|2012-09-12|

---

RU2519058C2|2014-06-10|

---

US8724914B2|2014-05-13|

---

US20120268465A1|2012-10-25|

---

AU2010313045A1|2012-06-14|

---

WO2011052117A1|2011-05-05|

---

RU2012121208A|2013-12-10|

---

AU2010313045B2|2013-08-29|

引用文献:

---

公开号 | 申请日 | 公开日 | 申请人 | 专利标题

---

---

US5081698A|1989-02-14|1992-01-14|Intel Corporation|Method and apparatus for graphics display data manipulation|

---

JP2798168B2|1993-01-27|1998-09-17|富士ゼロックス株式会社|Image coding device|

---

JPH07131658A|1993-10-29|1995-05-19|Canon Inc|Image communication equipment|

---

US6331856B1|1995-11-22|2001-12-18|Nintendo Co., Ltd.|Video game system with coprocessor providing high speed efficient 3D graphics and digital audio signal processing|

---

JP3102369B2|1997-01-31|2000-10-23|富士通株式会社|Data processing device|

---

SG64486A1|1997-03-27|1999-04-27|Sony Corp|Method and apparatus for information processing computer readable medium and authoring system|

---

US6556713B2|1997-07-31|2003-04-29|Canon Kabushiki Kaisha|Image processing apparatus and method and storage medium|

---

US6720971B1|1998-05-29|2004-04-13|Canon Kabushiki Kaisha|Image processing method and apparatus, and storage medium|

---

US6229926B1|1998-07-24|2001-05-08|Picsurf, Inc.|Memory saving wavelet-like image transform system and method for digital camera and other memory conservative applications|

---

US6535644B1|1999-07-01|2003-03-18|Koninklijke Philips Electronics N.V.|Hierarchical foveation based on wavelets|

---

CN1131637C|2000-10-13|2003-12-17|北京算通数字技术研究中心有限公司|Method of generating data stream index file and using said file accessing frame and shearing lens|

---

JP4467267B2|2002-09-06|2010-05-26|株式会社ソニー・コンピュータエンタテインメント|Image processing method, image processing apparatus, and image processing system|

---

US7454069B2|2002-11-20|2008-11-18|Ricoh Company, Ltd.|Image processing apparatus, image compressing apparatus, image decompressing apparatus, image processing method, image compressing method, image decompressing method, information processing apparatus, information processing method, program and recording medium|

---

US6990637B2|2003-10-23|2006-01-24|Microsoft Corporation|Graphical user interface for 3-dimensional view of a data collection based on an attribute of the data|

---

US7570832B2|2004-06-14|2009-08-04|Precoad Inc.|Image clean-up and pre-coding|

---

US7551182B2|2005-01-18|2009-06-23|Oculus Info Inc.|System and method for processing map data|

---

DE102006018689A1|2006-04-13|2007-10-25|Seereal Technologies S.A.|Method for rendering and generating computer-generated video holograms in real time|

---

JP5215945B2|2008-08-12|2013-06-19|株式会社ソニー・コンピュータエンタテインメント|Image processing device|

---

JP4744623B2|2009-07-01|2011-08-10|シャープ株式会社|Image compression method, image compression apparatus, image forming apparatus, computer program, and recording medium|JP5747378B2|2011-03-18|2015-07-15|株式会社日立国際電気|Image transfer system, image transfer method, image receiving apparatus, image transmitting apparatus, and image capturing apparatus|

---

US9324299B2|2011-09-09|2016-04-26|Microsoft Technology Licensing, Llc.|Atlasing and virtual surfaces|

---

US9135885B2|2011-11-30|2015-09-15|Adobe Systems Incorporated|Resolution independent client side rendering|

---

JP5548671B2|2011-12-27|2014-07-16|株式会社ソニー・コンピュータエンタテインメント|Image processing system, image providing server, information processing apparatus, and image processing method|

---

JP5722761B2|2011-12-27|2015-05-27|株式会社ソニー・コンピュータエンタテインメント|Video compression apparatus, image processing apparatus, video compression method, image processing method, and data structure of video compression file|

---

US9280956B2|2012-11-29|2016-03-08|Qualcomm Incorporated|Graphics memory load mask for graphics processing|

---

USD885418S1|2015-05-19|2020-05-26|Idea Connection Systems, Inc.|Display screen or portion thereof with graphical user interface|

---

JP6232940B2|2013-11-01|2017-11-22|富士ゼロックス株式会社|Image information processing apparatus and program|

---

CN104035751B|2014-06-20|2016-10-12|深圳市腾讯计算机系统有限公司|Data parallel processing method based on multi-graphics processor and device|

---

WO2016023999A2|2014-08-13|2016-02-18|King.Com Limited|Composing an image|

---

USD770468S1|2014-10-06|2016-11-01|National Comprehensive Cancer Network|Display screen or portion thereof with graphical user interface for clinical practice guidelines|

---

USD770469S1|2014-10-06|2016-11-01|National Comprehensive Cancer Network|Display screen or portion thereof with icon|

---

USD772889S1|2014-10-06|2016-11-29|National Comprehensive Cancer Network|Display screen or portion thereof with graphical user interface for clinical practice guidelines|

---

USD780768S1|2014-10-06|2017-03-07|National Comprehensive Cancer Network|Display screen or portion thereof with icon|

---

KR20160084147A|2015-01-05|2016-07-13|삼성전자주식회사|Image metadata managing method and apparatus|

---

US9489710B2|2015-02-10|2016-11-08|Qualcomm Incorporated|Hybrid rendering in graphics processing|

---

US10410398B2|2015-02-20|2019-09-10|Qualcomm Incorporated|Systems and methods for reducing memory bandwidth using low quality tiles|

---

RU2609076C2|2015-03-16|2017-01-30|Рамиль Ильдарович Хантимиров|Method and system for smart control over distribution of resources in cloud computing environments|

---

US10244241B2|2015-03-22|2019-03-26|Innova Plex, Inc.|Pyramidal file structure and method of use thereof|

---

GB2539509B|2015-06-19|2017-06-14|Advanced Risc Mach Ltd|Method of and apparatus for processing graphics|

---

CN105100826B|2015-07-17|2018-11-23|小米科技有限责任公司|Image processing method, apparatus and system|

---

US10535114B2|2015-08-18|2020-01-14|Nvidia Corporation|Controlling multi-pass rendering sequences in a cache tiling architecture|

---

US9990690B2|2015-09-21|2018-06-05|Qualcomm Incorporated|Efficient display processing with pre-fetching|

---

EP3362991A1|2015-10-17|2018-08-22|Arivis AG|Direct volume rendering in virtual and/or augmented reality|

---

TWI610292B|2015-11-19|2018-01-01|瑞鼎科技股份有限公司|Driving circuit and operating method thereof|

---

BR102015032565A2|2015-12-22|2017-06-27|Universidade Estadual De Campinas - Unicamp|METHOD OF INTERACTIVE VISUALIZATION OF A MAX-TREE TREE STRUCTURE AND USE OF THE SAME|

---

US10540007B2|2016-03-04|2020-01-21|Rockwell Collins, Inc.|Systems and methods for delivering imagery to head-worn display systems|

---

US10157498B2|2016-08-18|2018-12-18|Robert Bosch Gmbh|System and method for procedurally generated object distribution in regions of a three-dimensional virtual environment|

---

JP6702602B2|2016-08-25|2020-06-03|Ｎｅｃディスプレイソリューションズ株式会社|Self image diagnostic method, self image diagnostic program, display device, and self image diagnostic system|

---

JP6840506B2|2016-10-20|2021-03-10|キヤノン株式会社|Image processing device, image processing method, program|

---

US11222397B2|2016-12-23|2022-01-11|Qualcomm Incorporated|Foveated rendering in tiled architectures|

---

CN106991143B|2017-03-22|2019-07-19|苏州中科图新网络科技有限公司|Multi-layer image file, generation method and device, read method and device|

---

RU2677573C2|2017-04-05|2019-01-17|Общество С Ограниченной Ответственностью "Яндекс"|System and method of adding stylized properties to image|

---

US10885607B2|2017-06-01|2021-01-05|Qualcomm Incorporated|Storage for foveated rendering|

---

CN109388448B|2017-08-09|2020-08-04|京东方科技集团股份有限公司|Image display method, display system, and computer-readable storage medium|

---

GB2574361B|2017-12-18|2021-03-24|Advanced Risc Mach Ltd|Graphics Processing|

---

US10726610B2|2018-08-29|2020-07-28|Arm Limited|Efficient graphics processing using metadata|

---

US11049290B2|2019-03-21|2021-06-29|Adobe Inc.|Generation of a sequence of textures for video delivery|

法律状态:
2019-01-15| B06F| Objections, documents and/or translations needed after an examination request according [chapter 6.6 patent gazette]|

---

2019-07-30| B06U| Preliminary requirement: requests with searches performed by other patent offices: procedure suspended [chapter 6.21 patent gazette]|

---

2020-12-08| B09A| Decision: intention to grant [chapter 9.1 patent gazette]|

---

2021-02-02| B16A| Patent or certificate of addition of invention granted [chapter 16.1 patent gazette]|Free format text: PRAZO DE VALIDADE: 10 (DEZ) ANOS CONTADOS A PARTIR DE 02/02/2021, OBSERVADAS AS CONDICOES LEGAIS. |

优先权:

---

申请号 | 申请日 | 专利标题

---

JP2009245797A|JP5368254B2|2009-10-26|2009-10-26|Image file generation device, image processing device, image file generation method, image processing method, and data structure of image file|

---

JP2009245798A|JP5296656B2|2009-10-26|2009-10-26|Image processing apparatus and image processing method|

---

JP2009-245797|2009-10-26|

---

JP2009-245798|2009-10-26|

---

PCT/JP2010/004809|WO2011052117A1|2009-10-26|2010-07-29|Image file generation device, image processing device, image file generation method, image processing method, and data structure for image files|

---